2024-04-20T20:10:08.0069747Z Current runner version: '2.315.0' 2024-04-20T20:10:08.0078012Z Runner name: 'gh-ci-gcp-a100-11' 2024-04-20T20:10:08.0078965Z Runner group name: 'Default' 2024-04-20T20:10:08.0079965Z Machine name: 'gh-ci-gcp-a100-11' 2024-04-20T20:10:08.0085619Z ##[group]GITHUB_TOKEN Permissions 2024-04-20T20:10:08.0088204Z Actions: read 2024-04-20T20:10:08.0088832Z Attestations: read 2024-04-20T20:10:08.0089416Z Checks: read 2024-04-20T20:10:08.0090038Z Contents: read 2024-04-20T20:10:08.0090632Z Deployments: read 2024-04-20T20:10:08.0091218Z Discussions: read 2024-04-20T20:10:08.0091841Z Issues: read 2024-04-20T20:10:08.0092327Z Metadata: read 2024-04-20T20:10:08.0093063Z Packages: read 2024-04-20T20:10:08.0093666Z Pages: read 2024-04-20T20:10:08.0094212Z PullRequests: read 2024-04-20T20:10:08.0094780Z RepositoryProjects: read 2024-04-20T20:10:08.0095480Z SecurityEvents: read 2024-04-20T20:10:08.0096082Z Statuses: read 2024-04-20T20:10:08.0096602Z ##[endgroup] 2024-04-20T20:10:08.0100953Z Secret source: Actions 2024-04-20T20:10:08.0101949Z Prepare workflow directory 2024-04-20T20:10:08.3437088Z Prepare all required actions 2024-04-20T20:10:08.3649799Z Getting action download info 2024-04-20T20:10:08.5660783Z Download action repository 'pytorch/test-infra@main' (SHA:f2f9a0e2ac81ab590f98aa2985abf795143409a0) 2024-04-20T20:10:09.1961137Z Download action repository 'pytorch/pytorch@main' (SHA:afa78ad08cc748c25e7e82cec02cec4c97c7d3af) 2024-04-20T20:10:14.8198925Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-04-20T20:10:15.2853062Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-04-20T20:10:15.8526529Z Getting action download info 2024-04-20T20:10:15.9812784Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-04-20T20:10:16.4501407Z Getting action download info 2024-04-20T20:10:16.5728417Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-04-20T20:10:16.9570951Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (f20e3ae0c36146c962a5665018e9ad662a7cf211) 2024-04-20T20:10:16.9574130Z ##[group] Inputs 2024-04-20T20:10:16.9574739Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm80 2024-04-20T20:10:16.9581788Z test-matrix: {"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]} 2024-04-20T20:10:16.9590004Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:10:16.9591517Z sync-tag: 2024-04-20T20:10:16.9592552Z timeout-minutes: 720 2024-04-20T20:10:16.9593024Z use-gha: anything-non-empty-to-use-gha 2024-04-20T20:10:16.9595033Z dashboard-tag: training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true 2024-04-20T20:10:16.9596893Z s3-bucket: gha-artifacts 2024-04-20T20:10:16.9597311Z aws-role-to-assume: 2024-04-20T20:10:16.9597711Z ##[endgroup] 2024-04-20T20:10:16.9598880Z Complete job name: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:10:17.0333455Z A job started hook has been configured by the self-hosted runner administrator 2024-04-20T20:10:17.0615682Z ##[group]Run '/home/weiwangmeta/pre-job.sh' 2024-04-20T20:10:17.0648883Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:10:17.0649531Z ##[endgroup] 2024-04-20T20:10:17.1345445Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-04-20T20:10:17.1346158Z with: 2024-04-20T20:10:17.1346494Z submodules: recursive 2024-04-20T20:10:17.1346899Z fetch-depth: 0 2024-04-20T20:10:17.1347228Z env: 2024-04-20T20:10:17.1347559Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:10:17.1347984Z ##[endgroup] 2024-04-20T20:10:17.1578657Z ##[group]Run retry () { 2024-04-20T20:10:17.1579129Z retry () { 2024-04-20T20:10:17.1579735Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-04-20T20:10:17.1580425Z } 2024-04-20T20:10:17.1580772Z echo "${GITHUB_WORKSPACE}" 2024-04-20T20:10:17.1581338Z if [ -z "${NO_SUDO}" ]; then 2024-04-20T20:10:17.1581917Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-04-20T20:10:17.1582471Z else 2024-04-20T20:10:17.1582952Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-04-20T20:10:17.1583718Z fi 2024-04-20T20:10:17.1584296Z mkdir "${GITHUB_WORKSPACE}" 2024-04-20T20:10:17.1598193Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:10:17.1598854Z env: 2024-04-20T20:10:17.1599169Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:10:17.1599573Z NO_SUDO: 2024-04-20T20:10:17.1599914Z ##[endgroup] 2024-04-20T20:10:17.1680715Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2024-04-20T20:10:19.7769846Z ##[group]Run malfet/checkout@silent-checkout 2024-04-20T20:10:19.7770432Z with: 2024-04-20T20:10:19.7770814Z ref: f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:10:19.7771352Z fetch-depth: 0 2024-04-20T20:10:19.7771720Z submodules: recursive 2024-04-20T20:10:19.7772124Z quiet-checkout: true 2024-04-20T20:10:19.7772519Z repository: pytorch/pytorch 2024-04-20T20:10:19.7773292Z token: *** 2024-04-20T20:10:19.7773629Z ssh-strict: true 2024-04-20T20:10:19.7774010Z persist-credentials: true 2024-04-20T20:10:19.7774417Z clean: true 2024-04-20T20:10:19.7774813Z sparse-checkout-cone-mode: true 2024-04-20T20:10:19.7775272Z lfs: false 2024-04-20T20:10:19.7775619Z set-safe-directory: true 2024-04-20T20:10:19.7775999Z env: 2024-04-20T20:10:19.7776371Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:10:19.7776773Z ##[endgroup] 2024-04-20T20:10:19.9161646Z Syncing repository: pytorch/pytorch 2024-04-20T20:10:19.9163659Z ##[group]Getting Git version info 2024-04-20T20:10:19.9164661Z Working directory is '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch' 2024-04-20T20:10:19.9166659Z Unexpected error attempting to determine if executable file exists '/home/weiwangmeta/.local/bin/git': Error: EACCES: permission denied, stat '/home/weiwangmeta/.local/bin/git' 2024-04-20T20:10:19.9168390Z [command]/usr/bin/git version 2024-04-20T20:10:19.9168827Z git version 2.25.1 2024-04-20T20:10:19.9175807Z ##[endgroup] 2024-04-20T20:10:19.9193709Z Temporarily overriding HOME='/home/weiwangmeta/actions-runner/_work/_temp/ab39d958-12ec-462b-a272-da662a2f381a' before making global git config changes 2024-04-20T20:10:19.9195347Z Adding repository directory to the temporary git global config as a safe directory 2024-04-20T20:10:19.9200503Z [command]/usr/bin/git config --global --add safe.directory /home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2024-04-20T20:10:19.9249826Z Deleting the contents of '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch' 2024-04-20T20:10:19.9255394Z ##[group]Initializing the repository 2024-04-20T20:10:19.9259254Z [command]/usr/bin/git init /home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2024-04-20T20:10:19.9302428Z Initialized empty Git repository in /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/ 2024-04-20T20:10:19.9312217Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-04-20T20:10:19.9357316Z ##[endgroup] 2024-04-20T20:10:19.9358004Z ##[group]Disabling automatic garbage collection 2024-04-20T20:10:19.9360823Z [command]/usr/bin/git config --local gc.auto 0 2024-04-20T20:10:19.9397181Z ##[endgroup] 2024-04-20T20:10:19.9398087Z ##[group]Setting up auth 2024-04-20T20:10:19.9404309Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-04-20T20:10:19.9440657Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-04-20T20:10:19.9691966Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-04-20T20:10:19.9726598Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-04-20T20:10:19.9976122Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-20T20:10:20.0020464Z ##[endgroup] 2024-04-20T20:10:20.0021140Z ##[group]Fetching the repository 2024-04-20T20:10:20.0029828Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-04-20T20:10:25.2262975Z remote: Enumerating objects: 1097098 2024-04-20T20:10:25.2264147Z remote: Enumerating objects: 1099141, done. 2024-04-20T20:10:25.2264989Z remote: Counting objects: 0% (1/2043) 2024-04-20T20:10:25.2265601Z remote: Counting objects: 1% (21/2043) 2024-04-20T20:10:25.2266266Z remote: Counting objects: 2% (41/2043) 2024-04-20T20:10:25.2267357Z remote: Counting objects: 3% (62/2043) 2024-04-20T20:10:25.2268544Z remote: Counting objects: 4% (82/2043) 2024-04-20T20:10:25.2269480Z remote: Counting objects: 5% (103/2043) 2024-04-20T20:10:25.2270456Z remote: Counting objects: 6% (123/2043) 2024-04-20T20:10:25.2271670Z remote: Counting objects: 7% (144/2043) 2024-04-20T20:10:25.2272300Z remote: Counting objects: 8% (164/2043) 2024-04-20T20:10:25.2273416Z remote: Counting objects: 9% (184/2043) 2024-04-20T20:10:25.2274110Z remote: Counting objects: 10% (205/2043) 2024-04-20T20:10:25.2274709Z remote: Counting objects: 11% (225/2043) 2024-04-20T20:10:25.2275316Z remote: Counting objects: 12% (246/2043) 2024-04-20T20:10:25.2275997Z remote: Counting objects: 13% (266/2043) 2024-04-20T20:10:25.2276689Z remote: Counting objects: 14% (287/2043) 2024-04-20T20:10:25.2277282Z remote: Counting objects: 15% (307/2043) 2024-04-20T20:10:25.2278772Z remote: Counting objects: 16% (327/2043) 2024-04-20T20:10:25.2280004Z remote: Counting objects: 17% (348/2043) 2024-04-20T20:10:25.2281124Z remote: Counting objects: 18% (368/2043) 2024-04-20T20:10:25.2281973Z remote: Counting objects: 19% (389/2043) 2024-04-20T20:10:25.2282585Z remote: Counting objects: 20% (409/2043) 2024-04-20T20:10:25.2283532Z remote: Counting objects: 21% (430/2043) 2024-04-20T20:10:25.2284735Z remote: Counting objects: 22% (450/2043) 2024-04-20T20:10:25.2285841Z remote: Counting objects: 23% (470/2043) 2024-04-20T20:10:25.2286949Z remote: Counting objects: 24% (491/2043) 2024-04-20T20:10:25.2287600Z remote: Counting objects: 25% (511/2043) 2024-04-20T20:10:25.2288638Z remote: Counting objects: 26% (532/2043) 2024-04-20T20:10:25.2289244Z remote: Counting objects: 27% (552/2043) 2024-04-20T20:10:25.2290197Z remote: Counting objects: 28% (573/2043) 2024-04-20T20:10:25.2291176Z remote: Counting objects: 29% (593/2043) 2024-04-20T20:10:25.2291771Z remote: Counting objects: 30% (613/2043) 2024-04-20T20:10:25.2292804Z remote: Counting objects: 31% (634/2043) 2024-04-20T20:10:25.2293824Z remote: Counting objects: 32% (654/2043) 2024-04-20T20:10:25.2294434Z remote: Counting objects: 33% (675/2043) 2024-04-20T20:10:25.2295034Z remote: Counting objects: 34% (695/2043) 2024-04-20T20:10:25.2296180Z remote: Counting objects: 35% (716/2043) 2024-04-20T20:10:25.2296810Z remote: Counting objects: 36% (736/2043) 2024-04-20T20:10:25.2297417Z remote: Counting objects: 37% (756/2043) 2024-04-20T20:10:25.2298299Z remote: Counting objects: 38% (777/2043) 2024-04-20T20:10:25.2298897Z remote: Counting objects: 39% (797/2043) 2024-04-20T20:10:25.2299499Z remote: Counting objects: 40% (818/2043) 2024-04-20T20:10:25.2300100Z remote: Counting objects: 41% (838/2043) 2024-04-20T20:10:25.2300699Z remote: Counting objects: 42% (859/2043) 2024-04-20T20:10:25.2301281Z remote: Counting objects: 43% (879/2043) 2024-04-20T20:10:25.2301879Z remote: Counting objects: 44% (899/2043) 2024-04-20T20:10:25.2302479Z remote: Counting objects: 45% (920/2043) 2024-04-20T20:10:25.2303079Z remote: Counting objects: 46% (940/2043) 2024-04-20T20:10:25.2303665Z remote: Counting objects: 47% (961/2043) 2024-04-20T20:10:25.2304267Z remote: Counting objects: 48% (981/2043) 2024-04-20T20:10:25.2304881Z remote: Counting objects: 49% (1002/2043) 2024-04-20T20:10:25.2305508Z remote: Counting objects: 50% (1022/2043) 2024-04-20T20:10:25.2306170Z remote: Counting objects: 51% (1042/2043) 2024-04-20T20:10:25.2306805Z remote: Counting objects: 52% (1063/2043) 2024-04-20T20:10:25.2307428Z remote: Counting objects: 53% (1083/2043) 2024-04-20T20:10:25.2308054Z remote: Counting objects: 54% (1104/2043) 2024-04-20T20:10:25.2308666Z remote: Counting objects: 55% (1124/2043) 2024-04-20T20:10:25.2309286Z remote: Counting objects: 56% (1145/2043) 2024-04-20T20:10:25.2309906Z remote: Counting objects: 57% (1165/2043) 2024-04-20T20:10:25.2310527Z remote: Counting objects: 58% (1185/2043) 2024-04-20T20:10:25.2311137Z remote: Counting objects: 59% (1206/2043) 2024-04-20T20:10:25.2311761Z remote: Counting objects: 60% (1226/2043) 2024-04-20T20:10:25.2312379Z remote: Counting objects: 61% (1247/2043) 2024-04-20T20:10:25.2313000Z remote: Counting objects: 62% (1267/2043) 2024-04-20T20:10:25.2313602Z remote: Counting objects: 63% (1288/2043) 2024-04-20T20:10:25.2314227Z remote: Counting objects: 64% (1308/2043) 2024-04-20T20:10:25.2314848Z remote: Counting objects: 65% (1328/2043) 2024-04-20T20:10:25.2315469Z remote: Counting objects: 66% (1349/2043) 2024-04-20T20:10:25.2316079Z remote: Counting objects: 67% (1369/2043) 2024-04-20T20:10:25.2316737Z remote: Counting objects: 68% (1390/2043) 2024-04-20T20:10:25.2317356Z remote: Counting objects: 69% (1410/2043) 2024-04-20T20:10:25.2317978Z remote: Counting objects: 70% (1431/2043) 2024-04-20T20:10:25.2318582Z remote: Counting objects: 71% (1451/2043) 2024-04-20T20:10:25.2319202Z remote: Counting objects: 72% (1471/2043) 2024-04-20T20:10:25.2319823Z remote: Counting objects: 73% (1492/2043) 2024-04-20T20:10:25.2320447Z remote: Counting objects: 74% (1512/2043) 2024-04-20T20:10:25.2321058Z remote: Counting objects: 75% (1533/2043) 2024-04-20T20:10:25.2321676Z remote: Counting objects: 76% (1553/2043) 2024-04-20T20:10:25.2322559Z remote: Counting objects: 77% (1574/2043) 2024-04-20T20:10:25.2323186Z remote: Counting objects: 78% (1594/2043) 2024-04-20T20:10:25.2323792Z remote: Counting objects: 79% (1614/2043) 2024-04-20T20:10:25.2324415Z remote: Counting objects: 80% (1635/2043) 2024-04-20T20:10:25.2325035Z remote: Counting objects: 81% (1655/2043) 2024-04-20T20:10:25.2325658Z remote: Counting objects: 82% (1676/2043) 2024-04-20T20:10:25.2326302Z remote: Counting objects: 83% (1696/2043) 2024-04-20T20:10:25.2326921Z remote: Counting objects: 84% (1717/2043) 2024-04-20T20:10:25.2327542Z remote: Counting objects: 85% (1737/2043) 2024-04-20T20:10:25.2328297Z remote: Counting objects: 86% (1757/2043) 2024-04-20T20:10:25.2328909Z remote: Counting objects: 87% (1778/2043) 2024-04-20T20:10:25.2329527Z remote: Counting objects: 88% (1798/2043) 2024-04-20T20:10:25.2330148Z remote: Counting objects: 89% (1819/2043) 2024-04-20T20:10:25.2330770Z remote: Counting objects: 90% (1839/2043) 2024-04-20T20:10:25.2331381Z remote: Counting objects: 91% (1860/2043) 2024-04-20T20:10:25.2332001Z remote: Counting objects: 92% (1880/2043) 2024-04-20T20:10:25.2332621Z remote: Counting objects: 93% (1900/2043) 2024-04-20T20:10:25.2333420Z remote: Counting objects: 94% (1921/2043) 2024-04-20T20:10:25.2334061Z remote: Counting objects: 95% (1941/2043) 2024-04-20T20:10:25.2334671Z remote: Counting objects: 96% (1962/2043) 2024-04-20T20:10:25.2335296Z remote: Counting objects: 97% (1982/2043) 2024-04-20T20:10:25.2335917Z remote: Counting objects: 98% (2003/2043) 2024-04-20T20:10:25.2336577Z remote: Counting objects: 99% (2023/2043) 2024-04-20T20:10:25.2337189Z remote: Counting objects: 100% (2043/2043) 2024-04-20T20:10:25.2337851Z remote: Counting objects: 100% (2043/2043), done. 2024-04-20T20:10:25.2393617Z remote: Compressing objects: 0% (1/1085) 2024-04-20T20:10:25.2726701Z remote: Compressing objects: 1% (11/1085) 2024-04-20T20:10:25.2995304Z remote: Compressing objects: 2% (22/1085) 2024-04-20T20:10:25.3673740Z remote: Compressing objects: 3% (33/1085) 2024-04-20T20:10:25.5105729Z remote: Compressing objects: 4% (44/1085) 2024-04-20T20:10:25.6385561Z remote: Compressing objects: 5% (55/1085) 2024-04-20T20:10:25.7224213Z remote: Compressing objects: 6% (66/1085) 2024-04-20T20:10:25.7946662Z remote: Compressing objects: 7% (76/1085) 2024-04-20T20:10:25.8658817Z remote: Compressing objects: 8% (87/1085) 2024-04-20T20:10:25.9288121Z remote: Compressing objects: 9% (98/1085) 2024-04-20T20:10:25.9638062Z remote: Compressing objects: 10% (109/1085) 2024-04-20T20:10:25.9968817Z remote: Compressing objects: 11% (120/1085) 2024-04-20T20:10:26.0320660Z remote: Compressing objects: 12% (131/1085) 2024-04-20T20:10:26.0637142Z remote: Compressing objects: 13% (142/1085) 2024-04-20T20:10:26.0881580Z remote: Compressing objects: 14% (152/1085) 2024-04-20T20:10:26.1127611Z remote: Compressing objects: 15% (163/1085) 2024-04-20T20:10:26.1307642Z remote: Compressing objects: 16% (174/1085) 2024-04-20T20:10:26.1421494Z remote: Compressing objects: 17% (185/1085) 2024-04-20T20:10:26.1576260Z remote: Compressing objects: 18% (196/1085) 2024-04-20T20:10:26.1654102Z remote: Compressing objects: 19% (207/1085) 2024-04-20T20:10:26.1736226Z remote: Compressing objects: 20% (217/1085) 2024-04-20T20:10:26.1816170Z remote: Compressing objects: 21% (228/1085) 2024-04-20T20:10:26.1878108Z remote: Compressing objects: 22% (239/1085) 2024-04-20T20:10:26.1934422Z remote: Compressing objects: 23% (250/1085) 2024-04-20T20:10:26.1972661Z remote: Compressing objects: 24% (261/1085) 2024-04-20T20:10:26.1994696Z remote: Compressing objects: 25% (272/1085) 2024-04-20T20:10:26.2010152Z remote: Compressing objects: 26% (283/1085) 2024-04-20T20:10:26.2017659Z remote: Compressing objects: 27% (293/1085) 2024-04-20T20:10:26.2037170Z remote: Compressing objects: 28% (304/1085) 2024-04-20T20:10:26.2043943Z remote: Compressing objects: 29% (315/1085) 2024-04-20T20:10:26.2049652Z remote: Compressing objects: 30% (326/1085) 2024-04-20T20:10:26.2056673Z remote: Compressing objects: 31% (337/1085) 2024-04-20T20:10:26.2063515Z remote: Compressing objects: 32% (348/1085) 2024-04-20T20:10:26.2072658Z remote: Compressing objects: 33% (359/1085) 2024-04-20T20:10:26.2097454Z remote: Compressing objects: 34% (369/1085) 2024-04-20T20:10:26.2111385Z remote: Compressing objects: 35% (380/1085) 2024-04-20T20:10:26.2118234Z remote: Compressing objects: 36% (391/1085) 2024-04-20T20:10:26.2131518Z remote: Compressing objects: 37% (402/1085) 2024-04-20T20:10:26.2147274Z remote: Compressing objects: 38% (413/1085) 2024-04-20T20:10:26.2165500Z remote: Compressing objects: 39% (424/1085) 2024-04-20T20:10:26.2203472Z remote: Compressing objects: 40% (434/1085) 2024-04-20T20:10:26.2220609Z remote: Compressing objects: 41% (445/1085) 2024-04-20T20:10:26.2234074Z remote: Compressing objects: 42% (456/1085) 2024-04-20T20:10:26.2248021Z remote: Compressing objects: 43% (467/1085) 2024-04-20T20:10:26.2261349Z remote: Compressing objects: 44% (478/1085) 2024-04-20T20:10:26.2270292Z remote: Compressing objects: 45% (489/1085) 2024-04-20T20:10:26.2282946Z remote: Compressing objects: 46% (500/1085) 2024-04-20T20:10:26.2292101Z remote: Compressing objects: 47% (510/1085) 2024-04-20T20:10:26.2293604Z remote: Compressing objects: 47% (520/1085) 2024-04-20T20:10:26.2299844Z remote: Compressing objects: 48% (521/1085) 2024-04-20T20:10:26.2308207Z remote: Compressing objects: 49% (532/1085) 2024-04-20T20:10:26.2314435Z remote: Compressing objects: 50% (543/1085) 2024-04-20T20:10:26.2323612Z remote: Compressing objects: 51% (554/1085) 2024-04-20T20:10:26.2330555Z remote: Compressing objects: 52% (565/1085) 2024-04-20T20:10:26.2337070Z remote: Compressing objects: 53% (576/1085) 2024-04-20T20:10:26.2342634Z remote: Compressing objects: 54% (586/1085) 2024-04-20T20:10:26.2349162Z remote: Compressing objects: 55% (597/1085) 2024-04-20T20:10:26.2355997Z remote: Compressing objects: 56% (608/1085) 2024-04-20T20:10:26.2359349Z remote: Compressing objects: 57% (619/1085) 2024-04-20T20:10:26.2364165Z remote: Compressing objects: 58% (630/1085) 2024-04-20T20:10:26.2366261Z remote: Compressing objects: 59% (641/1085) 2024-04-20T20:10:26.2370331Z remote: Compressing objects: 60% (651/1085) 2024-04-20T20:10:26.2374400Z remote: Compressing objects: 61% (662/1085) 2024-04-20T20:10:26.2378097Z remote: Compressing objects: 62% (673/1085) 2024-04-20T20:10:26.2380608Z remote: Compressing objects: 63% (684/1085) 2024-04-20T20:10:26.2382905Z remote: Compressing objects: 64% (695/1085) 2024-04-20T20:10:26.2384362Z remote: Compressing objects: 65% (706/1085) 2024-04-20T20:10:26.2385821Z remote: Compressing objects: 66% (717/1085) 2024-04-20T20:10:26.2386521Z remote: Compressing objects: 67% (727/1085) 2024-04-20T20:10:26.2387481Z remote: Compressing objects: 68% (738/1085) 2024-04-20T20:10:26.2388731Z remote: Compressing objects: 69% (749/1085) 2024-04-20T20:10:26.2389368Z remote: Compressing objects: 70% (760/1085) 2024-04-20T20:10:26.2390031Z remote: Compressing objects: 71% (771/1085) 2024-04-20T20:10:26.2390680Z remote: Compressing objects: 72% (782/1085) 2024-04-20T20:10:26.2391324Z remote: Compressing objects: 73% (793/1085) 2024-04-20T20:10:26.2391971Z remote: Compressing objects: 74% (803/1085) 2024-04-20T20:10:26.2400269Z remote: Compressing objects: 75% (814/1085) 2024-04-20T20:10:26.2403290Z remote: Compressing objects: 76% (825/1085) 2024-04-20T20:10:26.2407911Z remote: Compressing objects: 77% (836/1085) 2024-04-20T20:10:26.2411312Z remote: Compressing objects: 78% (847/1085) 2024-04-20T20:10:26.2415148Z remote: Compressing objects: 79% (858/1085) 2024-04-20T20:10:26.2417975Z remote: Compressing objects: 80% (868/1085) 2024-04-20T20:10:26.2419843Z remote: Compressing objects: 81% (879/1085) 2024-04-20T20:10:26.2423379Z remote: Compressing objects: 82% (890/1085) 2024-04-20T20:10:26.2427376Z remote: Compressing objects: 83% (901/1085) 2024-04-20T20:10:26.2430072Z remote: Compressing objects: 84% (912/1085) 2024-04-20T20:10:26.2432505Z remote: Compressing objects: 85% (923/1085) 2024-04-20T20:10:26.2433600Z remote: Compressing objects: 86% (934/1085) 2024-04-20T20:10:26.2435070Z remote: Compressing objects: 87% (944/1085) 2024-04-20T20:10:26.2436797Z remote: Compressing objects: 88% (955/1085) 2024-04-20T20:10:26.2438514Z remote: Compressing objects: 89% (966/1085) 2024-04-20T20:10:26.2451778Z remote: Compressing objects: 90% (977/1085) 2024-04-20T20:10:26.2452625Z remote: Compressing objects: 91% (988/1085) 2024-04-20T20:10:26.2453693Z remote: Compressing objects: 92% (999/1085) 2024-04-20T20:10:26.2454347Z remote: Compressing objects: 93% (1010/1085) 2024-04-20T20:10:26.2455018Z remote: Compressing objects: 94% (1020/1085) 2024-04-20T20:10:26.2455683Z remote: Compressing objects: 95% (1031/1085) 2024-04-20T20:10:26.2456413Z remote: Compressing objects: 96% (1042/1085) 2024-04-20T20:10:26.2457068Z remote: Compressing objects: 97% (1053/1085) 2024-04-20T20:10:26.2457734Z remote: Compressing objects: 98% (1064/1085) 2024-04-20T20:10:26.2458395Z remote: Compressing objects: 99% (1075/1085) 2024-04-20T20:10:26.2459065Z remote: Compressing objects: 100% (1085/1085) 2024-04-20T20:10:26.2459763Z remote: Compressing objects: 100% (1085/1085), done. 2024-04-20T20:10:50.0335749Z remote: Total 1099141 (delta 1269), reused 1561 (delta 953), pack-reused 1097098 2024-04-20T20:11:28.4375074Z [command]/usr/bin/git rev-parse --verify --quiet f20e3ae0c36146c962a5665018e9ad662a7cf211^{object} 2024-04-20T20:11:28.4414475Z f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:11:28.4422794Z ##[endgroup] 2024-04-20T20:11:28.4423970Z ##[group]Determining the checkout info 2024-04-20T20:11:28.4427703Z ##[endgroup] 2024-04-20T20:11:28.4428789Z ##[group]Checking out the ref 2024-04-20T20:11:28.4430661Z [command]/usr/bin/git checkout --quiet --force f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:11:30.1339611Z ##[endgroup] 2024-04-20T20:11:30.1340328Z ##[group]Setting up auth for fetching submodules 2024-04-20T20:11:30.1346961Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-04-20T20:11:30.1402452Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-04-20T20:11:30.1439837Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-04-20T20:11:30.1477310Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-04-20T20:11:30.1509616Z ##[endgroup] 2024-04-20T20:11:30.1510236Z ##[group]Fetching submodules 2024-04-20T20:11:30.1515447Z [command]/usr/bin/git submodule sync --recursive 2024-04-20T20:11:30.1804682Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-04-20T20:11:30.2084711Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-04-20T20:11:30.2090444Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-04-20T20:11:30.2096810Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-04-20T20:11:30.2102687Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-04-20T20:11:30.2109542Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-04-20T20:11:30.2117013Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-04-20T20:11:30.2123289Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-04-20T20:11:30.2130267Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-04-20T20:11:30.2136352Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-04-20T20:11:30.2142404Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2024-04-20T20:11:30.2148226Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-04-20T20:11:30.2153217Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-04-20T20:11:30.2159791Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-04-20T20:11:30.2166426Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-04-20T20:11:30.2173052Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-04-20T20:11:30.2179192Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-04-20T20:11:30.2184799Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-04-20T20:11:30.2192235Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-04-20T20:11:30.2198993Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-04-20T20:11:30.2206161Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-04-20T20:11:30.2212691Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-04-20T20:11:30.2219551Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-04-20T20:11:30.2226138Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-04-20T20:11:30.2232612Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-04-20T20:11:30.2239172Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-04-20T20:11:30.2245666Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-04-20T20:11:30.2252142Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-04-20T20:11:30.2259824Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-04-20T20:11:30.2267116Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-04-20T20:11:30.2274016Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-04-20T20:11:30.2280764Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-04-20T20:11:30.2288401Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-04-20T20:11:30.2295342Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-04-20T20:11:30.2302615Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-04-20T20:11:30.2309903Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-04-20T20:11:30.2317102Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-04-20T20:11:30.2324150Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-04-20T20:11:30.2331582Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-04-20T20:11:30.2340064Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-04-20T20:11:30.2347919Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2024-04-20T20:11:30.2436682Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-04-20T20:11:30.7544571Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-04-20T20:11:31.1403971Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-04-20T20:11:31.4689357Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-04-20T20:11:31.9665075Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-04-20T20:11:32.4534435Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-04-20T20:11:35.6566160Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-04-20T20:11:51.4802256Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-04-20T20:11:52.1297061Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-04-20T20:11:53.0183753Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2024-04-20T20:11:54.8883216Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-04-20T20:11:56.3020064Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-04-20T20:11:58.3538955Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-04-20T20:12:03.8541507Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-04-20T20:12:05.5321393Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-04-20T20:12:07.5439790Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-04-20T20:12:09.4416587Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-04-20T20:12:09.8068238Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-04-20T20:12:10.4941857Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-04-20T20:12:11.0457133Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-04-20T20:12:12.5950214Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-04-20T20:12:13.1936336Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-04-20T20:12:13.5407984Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-04-20T20:12:14.2753001Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-04-20T20:12:16.0908409Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-04-20T20:12:17.0503125Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-04-20T20:12:17.8907647Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-04-20T20:12:18.4298901Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-04-20T20:12:25.9093178Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-04-20T20:12:29.5699181Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-04-20T20:12:30.3814956Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-04-20T20:12:30.7880562Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-04-20T20:12:41.9288351Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-04-20T20:12:42.2540493Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-04-20T20:12:42.6855629Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-04-20T20:12:43.8984397Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-04-20T20:12:44.4220410Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-04-20T20:12:45.3263768Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-04-20T20:12:47.6581014Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-04-20T20:12:48.2843769Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2024-04-20T20:12:51.4194650Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-04-20T20:12:51.4578726Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-04-20T20:12:51.4929409Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-04-20T20:12:51.5446346Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-04-20T20:12:51.5958620Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-04-20T20:12:51.6622274Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-04-20T20:12:52.7609165Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-04-20T20:12:52.8139561Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-04-20T20:12:52.9515844Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-04-20T20:12:53.0203737Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2024-04-20T20:12:53.0781842Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-04-20T20:12:53.6679314Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-04-20T20:12:53.9850262Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-04-20T20:12:54.0900080Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-04-20T20:12:54.0952768Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-04-20T20:12:54.0956856Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-04-20T20:12:54.0963331Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-04-20T20:12:54.0967751Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-04-20T20:12:54.0973574Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-04-20T20:12:54.1020438Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-04-20T20:12:55.4029887Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-04-20T20:12:56.3038604Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-04-20T20:12:58.3273084Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-04-20T20:12:59.8978274Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-04-20T20:13:00.4750173Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-04-20T20:13:00.6119692Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-04-20T20:13:01.0869925Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-04-20T20:13:01.1810669Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-04-20T20:13:01.2150428Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-04-20T20:13:01.3682642Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-04-20T20:13:01.4360501Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-04-20T20:13:01.4724684Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-04-20T20:13:01.5458787Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-04-20T20:13:01.5984799Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-04-20T20:13:01.6772218Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-04-20T20:13:01.7173042Z Submodule path 'third_party/ideep': checked out '8a6cc4e09dc509f04f83c085e38786b1fb44e14d' 2024-04-20T20:13:01.7221033Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-04-20T20:13:01.7263796Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-04-20T20:13:14.3731042Z Submodule path 'third_party/ideep/mkl-dnn': checked out '86e6af5974177e513fd3fee58425e1063e7f1361' 2024-04-20T20:13:14.4128995Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-04-20T20:13:14.4590452Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-04-20T20:13:14.5899997Z Submodule path 'third_party/kineto': checked out '47911e232609720f551b7a412076899d8a16a744' 2024-04-20T20:13:14.5954565Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-20T20:13:14.5958970Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-04-20T20:13:14.5964431Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-04-20T20:13:14.6011621Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-04-20T20:13:15.5682584Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-04-20T20:13:17.5126906Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-04-20T20:13:19.1456809Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-04-20T20:13:19.1510389Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-20T20:13:19.1515320Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-20T20:13:19.1522183Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-20T20:13:19.1541668Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-20T20:13:19.1543983Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-20T20:13:19.1549159Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-20T20:13:19.1551765Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-20T20:13:19.1555164Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-20T20:13:19.1599120Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-04-20T20:13:20.4373655Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-04-20T20:13:21.0628305Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-04-20T20:13:23.0095669Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-04-20T20:13:23.6506972Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-04-20T20:13:24.4118992Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-04-20T20:13:25.9282045Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-04-20T20:13:33.4815420Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-04-20T20:13:34.1419138Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-04-20T20:13:34.1842565Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-04-20T20:13:34.2484011Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-04-20T20:13:34.2851536Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-04-20T20:13:34.2901985Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-20T20:13:34.2947142Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-04-20T20:13:34.9763475Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-04-20T20:13:35.0180381Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-04-20T20:13:35.0864531Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-04-20T20:13:35.2269401Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-04-20T20:13:35.2650981Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-04-20T20:13:35.3256074Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-04-20T20:13:35.4129627Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-04-20T20:13:35.4794497Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-04-20T20:13:35.5307815Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-04-20T20:13:35.5733820Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-04-20T20:13:35.7162204Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-04-20T20:13:36.1111230Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-04-20T20:13:36.1181920Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-04-20T20:13:36.1187376Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-04-20T20:13:36.1255908Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-04-20T20:13:36.7354753Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-04-20T20:13:38.0012762Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-04-20T20:13:38.0637603Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-04-20T20:13:38.1097067Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-04-20T20:13:38.1144179Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-20T20:13:38.1188331Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-04-20T20:13:41.7662276Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-04-20T20:13:41.7720072Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-20T20:13:41.7726195Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-20T20:13:41.7781028Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-04-20T20:13:42.4102697Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-04-20T20:13:43.6480290Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-04-20T20:13:43.7495926Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-04-20T20:13:43.7546082Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-20T20:13:43.7591149Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-04-20T20:13:44.1917611Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-20T20:13:44.2281115Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-04-20T20:13:44.5572726Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-04-20T20:13:44.5630668Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-04-20T20:13:44.5635947Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-04-20T20:13:44.5686998Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-04-20T20:13:45.2835255Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-04-20T20:13:46.9010646Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-04-20T20:13:47.0051002Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-04-20T20:13:47.0408462Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-04-20T20:13:47.0809907Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-04-20T20:13:47.1432771Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-04-20T20:13:47.2035956Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-04-20T20:13:47.2778496Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-04-20T20:13:47.4285863Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-04-20T20:13:47.4839158Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-04-20T20:13:47.4888554Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-04-20T20:13:47.4893873Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-04-20T20:13:47.4900370Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-04-20T20:13:47.4906208Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-04-20T20:13:47.4952008Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-04-20T20:13:49.0286749Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-04-20T20:13:49.5003594Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-04-20T20:13:51.1122116Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-04-20T20:13:52.4122833Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-04-20T20:13:52.4487077Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-04-20T20:13:52.5506090Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-04-20T20:13:52.6024785Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-04-20T20:13:52.6074603Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-20T20:13:52.6119480Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-04-20T20:13:53.0256388Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-04-20T20:13:53.2141110Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2024-04-20T20:13:53.2239774Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-04-20T20:13:53.2527521Z Entering 'android/libs/fbjni' 2024-04-20T20:13:53.2568758Z Entering 'third_party/FP16' 2024-04-20T20:13:53.2608717Z Entering 'third_party/FXdiv' 2024-04-20T20:13:53.2649617Z Entering 'third_party/NNPACK' 2024-04-20T20:13:53.2689441Z Entering 'third_party/QNNPACK' 2024-04-20T20:13:53.2730565Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-20T20:13:53.2770796Z Entering 'third_party/XNNPACK' 2024-04-20T20:13:53.2831165Z Entering 'third_party/benchmark' 2024-04-20T20:13:53.2870719Z Entering 'third_party/cpuinfo' 2024-04-20T20:13:53.2911184Z Entering 'third_party/cub' 2024-04-20T20:13:53.2951638Z Entering 'third_party/cudnn_frontend' 2024-04-20T20:13:53.2992116Z Entering 'third_party/cutlass' 2024-04-20T20:13:53.3043540Z Entering 'third_party/eigen' 2024-04-20T20:13:53.3086355Z Entering 'third_party/fbgemm' 2024-04-20T20:13:53.3127005Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-20T20:13:53.3165119Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-20T20:13:53.3204377Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-20T20:13:53.3251114Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-20T20:13:53.3290191Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-20T20:13:53.3330862Z Entering 'third_party/flatbuffers' 2024-04-20T20:13:53.3373824Z Entering 'third_party/fmt' 2024-04-20T20:13:53.3414663Z Entering 'third_party/foxi' 2024-04-20T20:13:53.3454585Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-20T20:13:53.3494207Z Entering 'third_party/gloo' 2024-04-20T20:13:53.3534313Z Entering 'third_party/googletest' 2024-04-20T20:13:53.3573959Z Entering 'third_party/ideep' 2024-04-20T20:13:53.3613452Z Entering 'third_party/ideep/mkl-dnn' 2024-04-20T20:13:53.3660648Z Entering 'third_party/ios-cmake' 2024-04-20T20:13:53.3701046Z Entering 'third_party/ittapi' 2024-04-20T20:13:53.3741173Z Entering 'third_party/kineto' 2024-04-20T20:13:53.3781099Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-20T20:13:53.3819999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-20T20:13:53.3860235Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-20T20:13:53.3898725Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-20T20:13:53.3937957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-20T20:13:53.3975460Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-20T20:13:53.4016537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-20T20:13:53.4054656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-20T20:13:53.4093462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-20T20:13:53.4133660Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-20T20:13:53.4173500Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-20T20:13:53.4211461Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-20T20:13:53.4253441Z Entering 'third_party/mimalloc' 2024-04-20T20:13:53.4293999Z Entering 'third_party/nccl/nccl' 2024-04-20T20:13:53.4334727Z Entering 'third_party/neon2sse' 2024-04-20T20:13:53.4374201Z Entering 'third_party/nlohmann' 2024-04-20T20:13:53.4415586Z Entering 'third_party/onnx' 2024-04-20T20:13:53.4474933Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-20T20:13:53.4515083Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-20T20:13:53.4556908Z Entering 'third_party/onnx-tensorrt' 2024-04-20T20:13:53.4595962Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-20T20:13:53.4639975Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-20T20:13:53.4679545Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-20T20:13:53.4718242Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-20T20:13:53.4763174Z Entering 'third_party/pocketfft' 2024-04-20T20:13:53.4803222Z Entering 'third_party/protobuf' 2024-04-20T20:13:53.4846951Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-20T20:13:53.4885063Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-20T20:13:53.4926214Z Entering 'third_party/psimd' 2024-04-20T20:13:53.4966819Z Entering 'third_party/pthreadpool' 2024-04-20T20:13:53.5006474Z Entering 'third_party/pybind11' 2024-04-20T20:13:53.5047254Z Entering 'third_party/python-peachpy' 2024-04-20T20:13:53.5086704Z Entering 'third_party/sleef' 2024-04-20T20:13:53.5127274Z Entering 'third_party/tbb' 2024-04-20T20:13:53.5169400Z Entering 'third_party/tensorpipe' 2024-04-20T20:13:53.5210190Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-20T20:13:53.5248906Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-20T20:13:53.5286030Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-20T20:13:53.5323859Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-20T20:13:53.5361230Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-20T20:13:53.5402939Z Entering 'third_party/zstd' 2024-04-20T20:13:53.5461324Z ##[endgroup] 2024-04-20T20:13:53.5464577Z ##[group]Persisting credentials for submodules 2024-04-20T20:13:53.5483845Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-04-20T20:13:53.5762765Z Entering 'android/libs/fbjni' 2024-04-20T20:13:53.5809654Z Entering 'third_party/FP16' 2024-04-20T20:13:53.5855481Z Entering 'third_party/FXdiv' 2024-04-20T20:13:53.5902322Z Entering 'third_party/NNPACK' 2024-04-20T20:13:53.5948167Z Entering 'third_party/QNNPACK' 2024-04-20T20:13:53.5994283Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-20T20:13:53.6040803Z Entering 'third_party/XNNPACK' 2024-04-20T20:13:53.6106207Z Entering 'third_party/benchmark' 2024-04-20T20:13:53.6152317Z Entering 'third_party/cpuinfo' 2024-04-20T20:13:53.6198562Z Entering 'third_party/cub' 2024-04-20T20:13:53.6245098Z Entering 'third_party/cudnn_frontend' 2024-04-20T20:13:53.6290745Z Entering 'third_party/cutlass' 2024-04-20T20:13:53.6346122Z Entering 'third_party/eigen' 2024-04-20T20:13:53.6394698Z Entering 'third_party/fbgemm' 2024-04-20T20:13:53.6440496Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-20T20:13:53.6483978Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-20T20:13:53.6527963Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-20T20:13:53.6578776Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-20T20:13:53.6623223Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-20T20:13:53.6668679Z Entering 'third_party/flatbuffers' 2024-04-20T20:13:53.6716903Z Entering 'third_party/fmt' 2024-04-20T20:13:53.6763904Z Entering 'third_party/foxi' 2024-04-20T20:13:53.6809318Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-20T20:13:53.6855121Z Entering 'third_party/gloo' 2024-04-20T20:13:53.6901138Z Entering 'third_party/googletest' 2024-04-20T20:13:53.6946882Z Entering 'third_party/ideep' 2024-04-20T20:13:53.6990769Z Entering 'third_party/ideep/mkl-dnn' 2024-04-20T20:13:53.7043513Z Entering 'third_party/ios-cmake' 2024-04-20T20:13:53.7089128Z Entering 'third_party/ittapi' 2024-04-20T20:13:53.7135023Z Entering 'third_party/kineto' 2024-04-20T20:13:53.7180313Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-20T20:13:53.7223722Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-20T20:13:53.7268800Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-20T20:13:53.7311470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-20T20:13:53.7355115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-20T20:13:53.7397291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-20T20:13:53.7442776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-20T20:13:53.7487439Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-20T20:13:53.7532566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-20T20:13:53.7577548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-20T20:13:53.7623232Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-20T20:13:53.7667969Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-20T20:13:53.7713893Z Entering 'third_party/mimalloc' 2024-04-20T20:13:53.7759669Z Entering 'third_party/nccl/nccl' 2024-04-20T20:13:53.7804688Z Entering 'third_party/neon2sse' 2024-04-20T20:13:53.7849540Z Entering 'third_party/nlohmann' 2024-04-20T20:13:53.7896286Z Entering 'third_party/onnx' 2024-04-20T20:13:53.7960164Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-20T20:13:53.8005042Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-20T20:13:53.8051555Z Entering 'third_party/onnx-tensorrt' 2024-04-20T20:13:53.8097062Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-20T20:13:53.8147524Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-20T20:13:53.8191398Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-20T20:13:53.8235129Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-20T20:13:53.8285230Z Entering 'third_party/pocketfft' 2024-04-20T20:13:53.8330372Z Entering 'third_party/protobuf' 2024-04-20T20:13:53.8379173Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-20T20:13:53.8422889Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-20T20:13:53.8469348Z Entering 'third_party/psimd' 2024-04-20T20:13:53.8514338Z Entering 'third_party/pthreadpool' 2024-04-20T20:13:53.8560010Z Entering 'third_party/pybind11' 2024-04-20T20:13:53.8606028Z Entering 'third_party/python-peachpy' 2024-04-20T20:13:53.8651376Z Entering 'third_party/sleef' 2024-04-20T20:13:53.8696557Z Entering 'third_party/tbb' 2024-04-20T20:13:53.8744185Z Entering 'third_party/tensorpipe' 2024-04-20T20:13:53.8788944Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-20T20:13:53.8833408Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-20T20:13:53.8877616Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-20T20:13:53.8921623Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-20T20:13:53.8964945Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-20T20:13:53.9012213Z Entering 'third_party/zstd' 2024-04-20T20:13:53.9072609Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-04-20T20:13:53.9355927Z Entering 'android/libs/fbjni' 2024-04-20T20:13:53.9399274Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-04-20T20:13:53.9418757Z Entering 'third_party/FP16' 2024-04-20T20:13:53.9461114Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-04-20T20:13:53.9480179Z Entering 'third_party/FXdiv' 2024-04-20T20:13:53.9521379Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-04-20T20:13:53.9540367Z Entering 'third_party/NNPACK' 2024-04-20T20:13:53.9582545Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-04-20T20:13:53.9601825Z Entering 'third_party/QNNPACK' 2024-04-20T20:13:53.9643893Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-04-20T20:13:53.9663743Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-20T20:13:53.9705114Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-04-20T20:13:53.9724324Z Entering 'third_party/XNNPACK' 2024-04-20T20:13:53.9767578Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-04-20T20:13:53.9806453Z Entering 'third_party/benchmark' 2024-04-20T20:13:53.9849061Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-04-20T20:13:53.9869426Z Entering 'third_party/cpuinfo' 2024-04-20T20:13:53.9911885Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-04-20T20:13:53.9932276Z Entering 'third_party/cub' 2024-04-20T20:13:53.9973915Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2024-04-20T20:13:53.9992531Z Entering 'third_party/cudnn_frontend' 2024-04-20T20:13:54.0034845Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-04-20T20:13:54.0053779Z Entering 'third_party/cutlass' 2024-04-20T20:13:54.0095531Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-04-20T20:13:54.0123023Z Entering 'third_party/eigen' 2024-04-20T20:13:54.0164274Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-04-20T20:13:54.0187262Z Entering 'third_party/fbgemm' 2024-04-20T20:13:54.0229369Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-04-20T20:13:54.0247992Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-20T20:13:54.0289486Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-04-20T20:13:54.0307146Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-20T20:13:54.0347561Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-04-20T20:13:54.0366505Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-20T20:13:54.0406741Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-04-20T20:13:54.0434220Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-20T20:13:54.0479130Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-04-20T20:13:54.0494419Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-20T20:13:54.0535568Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-04-20T20:13:54.0555312Z Entering 'third_party/flatbuffers' 2024-04-20T20:13:54.0598390Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-04-20T20:13:54.0622139Z Entering 'third_party/fmt' 2024-04-20T20:13:54.0663664Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-04-20T20:13:54.0683118Z Entering 'third_party/foxi' 2024-04-20T20:13:54.0745736Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-04-20T20:13:54.0764741Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-20T20:13:54.0806971Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-04-20T20:13:54.0827335Z Entering 'third_party/gloo' 2024-04-20T20:13:54.0868822Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-04-20T20:13:54.0888426Z Entering 'third_party/googletest' 2024-04-20T20:13:54.0929993Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-04-20T20:13:54.0949736Z Entering 'third_party/ideep' 2024-04-20T20:13:54.0990510Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-04-20T20:13:54.1008173Z Entering 'third_party/ideep/mkl-dnn' 2024-04-20T20:13:54.1049931Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-04-20T20:13:54.1076648Z Entering 'third_party/ios-cmake' 2024-04-20T20:13:54.1119920Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-04-20T20:13:54.1139012Z Entering 'third_party/ittapi' 2024-04-20T20:13:54.1179984Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-04-20T20:13:54.1198622Z Entering 'third_party/kineto' 2024-04-20T20:13:54.1240862Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-04-20T20:13:54.1259243Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-20T20:13:54.1301717Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-04-20T20:13:54.1319510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-20T20:13:54.1362332Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-04-20T20:13:54.1393006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-20T20:13:54.1435533Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-04-20T20:13:54.1454159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-20T20:13:54.1495960Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-04-20T20:13:54.1513301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-20T20:13:54.1555007Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-04-20T20:13:54.1571864Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-20T20:13:54.1614044Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-04-20T20:13:54.1633860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-20T20:13:54.1675289Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-04-20T20:13:54.1693556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-20T20:13:54.1734958Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-04-20T20:13:54.1752989Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-20T20:13:54.1794886Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-04-20T20:13:54.1815026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-20T20:13:54.1857113Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-04-20T20:13:54.1877535Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-20T20:13:54.1918076Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-04-20T20:13:54.1936804Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-20T20:13:54.1977472Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-04-20T20:13:54.1996314Z Entering 'third_party/mimalloc' 2024-04-20T20:13:54.2038731Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-04-20T20:13:54.2058073Z Entering 'third_party/nccl/nccl' 2024-04-20T20:13:54.2100193Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-04-20T20:13:54.2119883Z Entering 'third_party/neon2sse' 2024-04-20T20:13:54.2161758Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-04-20T20:13:54.2180954Z Entering 'third_party/nlohmann' 2024-04-20T20:13:54.2222646Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-04-20T20:13:54.2243027Z Entering 'third_party/onnx' 2024-04-20T20:13:54.2284381Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-04-20T20:13:54.2322004Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-20T20:13:54.2364567Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-20T20:13:54.2383596Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-20T20:13:54.2425685Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-20T20:13:54.2446434Z Entering 'third_party/onnx-tensorrt' 2024-04-20T20:13:54.2487696Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-04-20T20:13:54.2505906Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-20T20:13:54.2547292Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-04-20T20:13:54.2570371Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-20T20:13:54.2612338Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-04-20T20:13:54.2631389Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-20T20:13:54.2673458Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-04-20T20:13:54.2690772Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-20T20:13:54.2733078Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-20T20:13:54.2756491Z Entering 'third_party/pocketfft' 2024-04-20T20:13:54.2798306Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-04-20T20:13:54.2817312Z Entering 'third_party/protobuf' 2024-04-20T20:13:54.2860402Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-04-20T20:13:54.2882855Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-20T20:13:54.2924639Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-04-20T20:13:54.2943002Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-20T20:13:54.2983855Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-04-20T20:13:54.3004132Z Entering 'third_party/psimd' 2024-04-20T20:13:54.3045780Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-04-20T20:13:54.3064998Z Entering 'third_party/pthreadpool' 2024-04-20T20:13:54.3106882Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-04-20T20:13:54.3125571Z Entering 'third_party/pybind11' 2024-04-20T20:13:54.3167283Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-04-20T20:13:54.3186449Z Entering 'third_party/python-peachpy' 2024-04-20T20:13:54.3226799Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-04-20T20:13:54.3246248Z Entering 'third_party/sleef' 2024-04-20T20:13:54.3286936Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-04-20T20:13:54.3306061Z Entering 'third_party/tbb' 2024-04-20T20:13:54.3347033Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-04-20T20:13:54.3367763Z Entering 'third_party/tensorpipe' 2024-04-20T20:13:54.3409277Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-04-20T20:13:54.3427844Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-20T20:13:54.3468439Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-04-20T20:13:54.3486622Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-20T20:13:54.3527223Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-04-20T20:13:54.3545641Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-20T20:13:54.3586502Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-04-20T20:13:54.3604698Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-20T20:13:54.3645256Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-04-20T20:13:54.3662884Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-20T20:13:54.3703516Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-04-20T20:13:54.3724671Z Entering 'third_party/zstd' 2024-04-20T20:13:54.3765667Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2024-04-20T20:13:54.4107743Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-04-20T20:13:54.4389779Z Entering 'android/libs/fbjni' 2024-04-20T20:13:54.4431077Z Entering 'third_party/FP16' 2024-04-20T20:13:54.4471877Z Entering 'third_party/FXdiv' 2024-04-20T20:13:54.4514067Z Entering 'third_party/NNPACK' 2024-04-20T20:13:54.4556701Z Entering 'third_party/QNNPACK' 2024-04-20T20:13:54.4598174Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-20T20:13:54.4639770Z Entering 'third_party/XNNPACK' 2024-04-20T20:13:54.4700975Z Entering 'third_party/benchmark' 2024-04-20T20:13:54.4741831Z Entering 'third_party/cpuinfo' 2024-04-20T20:13:54.4783904Z Entering 'third_party/cub' 2024-04-20T20:13:54.4825918Z Entering 'third_party/cudnn_frontend' 2024-04-20T20:13:54.4866615Z Entering 'third_party/cutlass' 2024-04-20T20:13:54.4916857Z Entering 'third_party/eigen' 2024-04-20T20:13:54.4960998Z Entering 'third_party/fbgemm' 2024-04-20T20:13:54.5001528Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-20T20:13:54.5041020Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-20T20:13:54.5082029Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-20T20:13:54.5128912Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-20T20:13:54.5168107Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-20T20:13:54.5208067Z Entering 'third_party/flatbuffers' 2024-04-20T20:13:54.5251455Z Entering 'third_party/fmt' 2024-04-20T20:13:54.5292025Z Entering 'third_party/foxi' 2024-04-20T20:13:54.5332128Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-20T20:13:54.5372433Z Entering 'third_party/gloo' 2024-04-20T20:13:54.5413293Z Entering 'third_party/googletest' 2024-04-20T20:13:54.5454124Z Entering 'third_party/ideep' 2024-04-20T20:13:54.5493226Z Entering 'third_party/ideep/mkl-dnn' 2024-04-20T20:13:54.5541280Z Entering 'third_party/ios-cmake' 2024-04-20T20:13:54.5582499Z Entering 'third_party/ittapi' 2024-04-20T20:13:54.5622987Z Entering 'third_party/kineto' 2024-04-20T20:13:54.5664345Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-20T20:13:54.5702697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-20T20:13:54.5743020Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-20T20:13:54.5781693Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-20T20:13:54.5820969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-20T20:13:54.5859841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-20T20:13:54.5900499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-20T20:13:54.5939699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-20T20:13:54.5978405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-20T20:13:54.6018799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-20T20:13:54.6060208Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-20T20:13:54.6101525Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-20T20:13:54.6141757Z Entering 'third_party/mimalloc' 2024-04-20T20:13:54.6183073Z Entering 'third_party/nccl/nccl' 2024-04-20T20:13:54.6224045Z Entering 'third_party/neon2sse' 2024-04-20T20:13:54.6265534Z Entering 'third_party/nlohmann' 2024-04-20T20:13:54.6308276Z Entering 'third_party/onnx' 2024-04-20T20:13:54.6368654Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-20T20:13:54.6408672Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-20T20:13:54.6451024Z Entering 'third_party/onnx-tensorrt' 2024-04-20T20:13:54.6490325Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-20T20:13:54.6535018Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-20T20:13:54.6574824Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-20T20:13:54.6612715Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-20T20:13:54.6659276Z Entering 'third_party/pocketfft' 2024-04-20T20:13:54.6698086Z Entering 'third_party/protobuf' 2024-04-20T20:13:54.6741881Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-20T20:13:54.6781783Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-20T20:13:54.6822824Z Entering 'third_party/psimd' 2024-04-20T20:13:54.6862598Z Entering 'third_party/pthreadpool' 2024-04-20T20:13:54.6902930Z Entering 'third_party/pybind11' 2024-04-20T20:13:54.6943254Z Entering 'third_party/python-peachpy' 2024-04-20T20:13:54.6983266Z Entering 'third_party/sleef' 2024-04-20T20:13:54.7023184Z Entering 'third_party/tbb' 2024-04-20T20:13:54.7065417Z Entering 'third_party/tensorpipe' 2024-04-20T20:13:54.7105567Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-20T20:13:54.7143834Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-20T20:13:54.7182102Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-20T20:13:54.7220287Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-20T20:13:54.7257831Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-20T20:13:54.7299510Z Entering 'third_party/zstd' 2024-04-20T20:13:54.7354904Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-04-20T20:13:54.7640173Z Entering 'android/libs/fbjni' 2024-04-20T20:13:54.7679520Z Entering 'third_party/FP16' 2024-04-20T20:13:54.7718785Z Entering 'third_party/FXdiv' 2024-04-20T20:13:54.7758917Z Entering 'third_party/NNPACK' 2024-04-20T20:13:54.7798948Z Entering 'third_party/QNNPACK' 2024-04-20T20:13:54.7839526Z Entering 'third_party/VulkanMemoryAllocator' 2024-04-20T20:13:54.7879112Z Entering 'third_party/XNNPACK' 2024-04-20T20:13:54.7939008Z Entering 'third_party/benchmark' 2024-04-20T20:13:54.7979936Z Entering 'third_party/cpuinfo' 2024-04-20T20:13:54.8020827Z Entering 'third_party/cub' 2024-04-20T20:13:54.8060989Z Entering 'third_party/cudnn_frontend' 2024-04-20T20:13:54.8101272Z Entering 'third_party/cutlass' 2024-04-20T20:13:54.8150435Z Entering 'third_party/eigen' 2024-04-20T20:13:54.8193048Z Entering 'third_party/fbgemm' 2024-04-20T20:13:54.8232456Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-04-20T20:13:54.8270490Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-04-20T20:13:54.8309658Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-04-20T20:13:54.8355683Z Entering 'third_party/fbgemm/third_party/googletest' 2024-04-20T20:13:54.8393856Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-04-20T20:13:54.8435173Z Entering 'third_party/flatbuffers' 2024-04-20T20:13:54.8478776Z Entering 'third_party/fmt' 2024-04-20T20:13:54.8518934Z Entering 'third_party/foxi' 2024-04-20T20:13:54.8559121Z Entering 'third_party/gemmlowp/gemmlowp' 2024-04-20T20:13:54.8599038Z Entering 'third_party/gloo' 2024-04-20T20:13:54.8639713Z Entering 'third_party/googletest' 2024-04-20T20:13:54.8680551Z Entering 'third_party/ideep' 2024-04-20T20:13:54.8719434Z Entering 'third_party/ideep/mkl-dnn' 2024-04-20T20:13:54.8766911Z Entering 'third_party/ios-cmake' 2024-04-20T20:13:54.8806072Z Entering 'third_party/ittapi' 2024-04-20T20:13:54.8845993Z Entering 'third_party/kineto' 2024-04-20T20:13:54.8885758Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-04-20T20:13:54.8925117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-04-20T20:13:54.8966516Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-04-20T20:13:54.9005702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-04-20T20:13:54.9044613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-04-20T20:13:54.9082605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-04-20T20:13:54.9125235Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-04-20T20:13:54.9164454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-04-20T20:13:54.9203813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-04-20T20:13:54.9243751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-04-20T20:13:54.9284560Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-04-20T20:13:54.9323154Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-04-20T20:13:54.9363226Z Entering 'third_party/mimalloc' 2024-04-20T20:13:54.9404054Z Entering 'third_party/nccl/nccl' 2024-04-20T20:13:54.9445379Z Entering 'third_party/neon2sse' 2024-04-20T20:13:54.9484896Z Entering 'third_party/nlohmann' 2024-04-20T20:13:54.9527367Z Entering 'third_party/onnx' 2024-04-20T20:13:54.9590646Z Entering 'third_party/onnx/third_party/benchmark' 2024-04-20T20:13:54.9630577Z Entering 'third_party/onnx/third_party/pybind11' 2024-04-20T20:13:54.9673930Z Entering 'third_party/onnx-tensorrt' 2024-04-20T20:13:54.9714283Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-04-20T20:13:54.9759149Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-04-20T20:13:54.9799175Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-04-20T20:13:54.9839560Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-04-20T20:13:54.9884028Z Entering 'third_party/pocketfft' 2024-04-20T20:13:54.9923344Z Entering 'third_party/protobuf' 2024-04-20T20:13:54.9967045Z Entering 'third_party/protobuf/third_party/benchmark' 2024-04-20T20:13:55.0005715Z Entering 'third_party/protobuf/third_party/googletest' 2024-04-20T20:13:55.0046437Z Entering 'third_party/psimd' 2024-04-20T20:13:55.0087487Z Entering 'third_party/pthreadpool' 2024-04-20T20:13:55.0127240Z Entering 'third_party/pybind11' 2024-04-20T20:13:55.0167938Z Entering 'third_party/python-peachpy' 2024-04-20T20:13:55.0208924Z Entering 'third_party/sleef' 2024-04-20T20:13:55.0249669Z Entering 'third_party/tbb' 2024-04-20T20:13:55.0292774Z Entering 'third_party/tensorpipe' 2024-04-20T20:13:55.0332748Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-04-20T20:13:55.0371146Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-04-20T20:13:55.0409287Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-04-20T20:13:55.0450928Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-04-20T20:13:55.0489887Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-04-20T20:13:55.0533816Z Entering 'third_party/zstd' 2024-04-20T20:13:55.0587134Z ##[endgroup] 2024-04-20T20:13:55.0637290Z [command]/usr/bin/git log -1 --format='%H' 2024-04-20T20:13:55.0669792Z 'f20e3ae0c36146c962a5665018e9ad662a7cf211' 2024-04-20T20:13:55.0961740Z Prepare all required actions 2024-04-20T20:13:55.0962478Z Getting action download info 2024-04-20T20:13:55.3408372Z ##[group]Run ./.github/actions/setup-linux 2024-04-20T20:13:55.3408893Z env: 2024-04-20T20:13:55.3409215Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:55.3409617Z ##[endgroup] 2024-04-20T20:13:55.3483446Z ##[group]Run set -euo pipefail 2024-04-20T20:13:55.3483974Z set -euo pipefail 2024-04-20T20:13:55.3484422Z function get_ec2_metadata() { 2024-04-20T20:13:55.3485067Z  # Pulled from instance metadata endpoint for EC2 2024-04-20T20:13:55.3486185Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-04-20T20:13:55.3487256Z  category=$1 2024-04-20T20:13:55.3487873Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-04-20T20:13:55.3488658Z  runner_name_str=gh-ci-gcp-a100-11 2024-04-20T20:13:55.3489264Z  if [[ -f /.inarc ]]; then 2024-04-20T20:13:55.3489839Z  echo "ARC Runner, no info on ec2 metadata" 2024-04-20T20:13:55.3490528Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-04-20T20:13:55.3491375Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-04-20T20:13:55.3492115Z  else 2024-04-20T20:13:55.3492687Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-04-20T20:13:55.3493536Z  fi 2024-04-20T20:13:55.3493862Z } 2024-04-20T20:13:55.3494281Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-04-20T20:13:55.3494988Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-04-20T20:13:55.3495791Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-04-20T20:13:55.3496537Z echo "system info $(uname -a)" 2024-04-20T20:13:55.3510844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:55.3511449Z env: 2024-04-20T20:13:55.3511782Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:55.3512170Z ##[endgroup] 2024-04-20T20:13:55.3572797Z ami-id: Runner is from Google Cloud Platform, No info on ec2 metadata 2024-04-20T20:13:55.3579227Z instance-id: Runner is from Google Cloud Platform, No info on ec2 metadata 2024-04-20T20:13:55.3584364Z instance-type: Runner is from Google Cloud Platform, No info on ec2 metadata 2024-04-20T20:13:55.3596153Z system info Linux gh-ci-gcp-a100-11 5.15.0-1041-gcp #49~20.04.1-Ubuntu SMP Tue Aug 29 06:49:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux 2024-04-20T20:13:55.3620892Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-20T20:13:55.3622058Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-04-20T20:13:55.3635102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:55.3635727Z env: 2024-04-20T20:13:55.3636121Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:55.3636541Z ##[endgroup] 2024-04-20T20:13:55.3729663Z ##[group]Run if systemctl is-active --quiet docker; then 2024-04-20T20:13:55.3730387Z if systemctl is-active --quiet docker; then 2024-04-20T20:13:55.3731110Z  echo "Docker daemon is running..."; 2024-04-20T20:13:55.3731646Z else 2024-04-20T20:13:55.3732226Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-04-20T20:13:55.3733233Z fi 2024-04-20T20:13:55.3745384Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:55.3745990Z env: 2024-04-20T20:13:55.3746310Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:55.3746882Z ##[endgroup] 2024-04-20T20:13:55.3813350Z Docker daemon is running... 2024-04-20T20:13:55.3869800Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-20T20:13:55.3870487Z with: 2024-04-20T20:13:55.3870788Z shell: bash 2024-04-20T20:13:55.3871115Z timeout_minutes: 5 2024-04-20T20:13:55.3871487Z max_attempts: 3 2024-04-20T20:13:55.3872075Z retry_wait_seconds: 30 2024-04-20T20:13:55.3873860Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-04-20T20:13:55.3875726Z polling_interval_seconds: 1 2024-04-20T20:13:55.3876213Z warning_on_retry: true 2024-04-20T20:13:55.3876624Z continue_on_error: false 2024-04-20T20:13:55.3877008Z env: 2024-04-20T20:13:55.3877317Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:55.3877717Z AWS_RETRY_MODE: standard 2024-04-20T20:13:55.3878130Z AWS_MAX_ATTEMPTS: 5 2024-04-20T20:13:55.3878529Z AWS_DEFAULT_REGION: us-east-1 2024-04-20T20:13:55.3878949Z ##[endgroup] 2024-04-20T20:13:57.5815708Z WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json. 2024-04-20T20:13:57.5816762Z Configure a credential helper to remove this warning. See 2024-04-20T20:13:57.5818093Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-20T20:13:57.5818770Z 2024-04-20T20:13:57.5818926Z Login Succeeded 2024-04-20T20:13:58.4600663Z Command completed after 1 attempt(s). 2024-04-20T20:13:58.4670033Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-20T20:13:58.4670932Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-20T20:13:58.4671817Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-04-20T20:13:58.4685459Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:58.4686069Z env: 2024-04-20T20:13:58.4686396Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:58.4686805Z ##[endgroup] 2024-04-20T20:13:58.4805078Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-04-20T20:13:58.4806106Z # ignore expansion of "docker ps -q" since it could be empty 2024-04-20T20:13:58.4806835Z # shellcheck disable=SC2046 2024-04-20T20:13:58.4807408Z docker stop $(docker ps -q) || true 2024-04-20T20:13:58.4807981Z # Prune all of the docker images 2024-04-20T20:13:58.4808505Z docker system prune -af 2024-04-20T20:13:58.4821028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:58.4821642Z env: 2024-04-20T20:13:58.4821967Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:58.4822376Z ##[endgroup] 2024-04-20T20:13:58.5245323Z "docker stop" requires at least 1 argument. 2024-04-20T20:13:58.5246531Z See 'docker stop --help'. 2024-04-20T20:13:58.5246819Z 2024-04-20T20:13:58.5247082Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-04-20T20:13:58.5247583Z 2024-04-20T20:13:58.5247751Z Stop one or more running containers 2024-04-20T20:13:58.5473088Z Total reclaimed space: 0B 2024-04-20T20:13:58.5517999Z ##[group]Run set +e 2024-04-20T20:13:58.5518420Z set +e 2024-04-20T20:13:58.5518771Z set -x 2024-04-20T20:13:58.5519102Z  2024-04-20T20:13:58.5519499Z PT_DOMAIN=download.pytorch.org 2024-04-20T20:13:58.5520470Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-04-20T20:13:58.5521888Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-04-20T20:13:58.5522832Z # one is returned at random 2024-04-20T20:13:58.5523486Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-04-20T20:13:58.5524097Z  2024-04-20T20:13:58.5524459Z if [ -z "${RESOLVED_IP}" ]; then 2024-04-20T20:13:58.5525214Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-04-20T20:13:58.5526380Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-04-20T20:13:58.5527045Z  2024-04-20T20:13:58.5527423Z  if [ -z "${RESOLVED_IP}" ]; then 2024-04-20T20:13:58.5528093Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-04-20T20:13:58.5528844Z  exit 1 2024-04-20T20:13:58.5529199Z  fi 2024-04-20T20:13:58.5529520Z fi 2024-04-20T20:13:58.5529835Z  2024-04-20T20:13:58.5530250Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-04-20T20:13:58.5530865Z  # Clean up any old records first 2024-04-20T20:13:58.5531514Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-04-20T20:13:58.5532056Z fi 2024-04-20T20:13:58.5532374Z  2024-04-20T20:13:58.5533058Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-04-20T20:13:58.5533819Z cat /etc/hosts 2024-04-20T20:13:58.5546671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:58.5547281Z env: 2024-04-20T20:13:58.5547601Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:58.5547995Z ##[endgroup] 2024-04-20T20:13:58.5582984Z + PT_DOMAIN=download.pytorch.org 2024-04-20T20:13:58.5589705Z ++ dig -4 +short download.pytorch.org 2024-04-20T20:13:58.5590971Z ++ tail -n1 2024-04-20T20:13:58.5679455Z + RESOLVED_IP=13.226.22.44 2024-04-20T20:13:58.5680064Z + '[' -z 13.226.22.44 ']' 2024-04-20T20:13:58.5680599Z + grep -r download.pytorch.org /etc/hosts 2024-04-20T20:13:58.5691327Z 13.226.22.44 download.pytorch.org 2024-04-20T20:13:58.5693160Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-04-20T20:13:58.5799386Z + echo '13.226.22.44 download.pytorch.org' 2024-04-20T20:13:58.5799992Z + sudo tee -a /etc/hosts 2024-04-20T20:13:58.5876723Z 13.226.22.44 download.pytorch.org 2024-04-20T20:13:58.5883594Z + cat /etc/hosts 2024-04-20T20:13:58.5891499Z 127.0.0.1 localhost 2024-04-20T20:13:58.5891855Z 2024-04-20T20:13:58.5892299Z # The following lines are desirable for IPv6 capable hosts 2024-04-20T20:13:58.5907363Z ::1 ip6-localhost ip6-loopback 2024-04-20T20:13:58.5907937Z fe00::0 ip6-localnet 2024-04-20T20:13:58.5908397Z ff00::0 ip6-mcastprefix 2024-04-20T20:13:58.5908822Z ff02::1 ip6-allnodes 2024-04-20T20:13:58.5909243Z ff02::2 ip6-allrouters 2024-04-20T20:13:58.5909686Z ff02::3 ip6-allhosts 2024-04-20T20:13:58.5910161Z 169.254.169.254 metadata.google.internal metadata 2024-04-20T20:13:58.5910737Z 13.226.22.44 download.pytorch.org 2024-04-20T20:13:58.6104498Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-04-20T20:13:58.6105242Z with: 2024-04-20T20:13:58.6106714Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6108319Z docker-build-dir: .ci/docker 2024-04-20T20:13:58.6108776Z working-directory: . 2024-04-20T20:13:58.6109361Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:13:58.6109999Z force-push: false 2024-04-20T20:13:58.6110348Z env: 2024-04-20T20:13:58.6110666Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:58.6111070Z ##[endgroup] 2024-04-20T20:13:58.6132468Z ##[group]Run set -ex 2024-04-20T20:13:58.6133216Z set -ex 2024-04-20T20:13:58.6133581Z  2024-04-20T20:13:58.6134270Z # If the docker build directory or the build script doesn't exist, the action will 2024-04-20T20:13:58.6135516Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-04-20T20:13:58.6136563Z # job could then download the pre-built image as usual 2024-04-20T20:13:58.6137493Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-04-20T20:13:58.6138328Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6139107Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6140003Z  2024-04-20T20:13:58.6140609Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-04-20T20:13:58.6141389Z  exit 0 2024-04-20T20:13:58.6141743Z else 2024-04-20T20:13:58.6142173Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6142701Z fi 2024-04-20T20:13:58.6143013Z  2024-04-20T20:13:58.6143576Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-04-20T20:13:58.6144645Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-04-20T20:13:58.6145598Z  # use it as it is, but first let's extract the tag 2024-04-20T20:13:58.6146509Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-04-20T20:13:58.6147410Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6148278Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6148957Z else 2024-04-20T20:13:58.6149476Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-04-20T20:13:58.6150292Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6151400Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6152327Z fi 2024-04-20T20:13:58.6164964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:58.6165557Z env: 2024-04-20T20:13:58.6165884Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:58.6166362Z REPO_NAME: pytorch 2024-04-20T20:13:58.6167828Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6169381Z DOCKER_BUILD_DIR: .ci/docker 2024-04-20T20:13:58.6169985Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:13:58.6170621Z ##[endgroup] 2024-04-20T20:13:58.6205523Z + [[ ! -d .ci/docker ]] 2024-04-20T20:13:58.6206021Z + [[ ! -f .ci/docker/build.sh ]] 2024-04-20T20:13:58.6206527Z + echo skip=false 2024-04-20T20:13:58.6209435Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-04-20T20:13:58.6215815Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6217423Z ++ awk -F '[:,]' '{print $2}' 2024-04-20T20:13:58.6239053Z + DOCKER_TAG=47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6239864Z + echo docker-tag=47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6242163Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6274237Z ##[group]Run set +e 2024-04-20T20:13:58.6274686Z set +e 2024-04-20T20:13:58.6275034Z set -x 2024-04-20T20:13:58.6275354Z  2024-04-20T20:13:58.6275674Z login() { 2024-04-20T20:13:58.6276573Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-20T20:13:58.6277467Z } 2024-04-20T20:13:58.6277767Z  2024-04-20T20:13:58.6278085Z retry () { 2024-04-20T20:13:58.6278542Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-20T20:13:58.6279074Z } 2024-04-20T20:13:58.6279373Z  2024-04-20T20:13:58.6279737Z retry login "${DOCKER_REGISTRY}" 2024-04-20T20:13:58.6280232Z  2024-04-20T20:13:58.6280783Z # Check if image already exists, if it does then skip building it 2024-04-20T20:13:58.6281846Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-04-20T20:13:58.6282457Z  exit 0 2024-04-20T20:13:58.6282804Z fi 2024-04-20T20:13:58.6283103Z  2024-04-20T20:13:58.6283704Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-04-20T20:13:58.6284739Z # be empty. The default action would be to continue rebuild the image 2024-04-20T20:13:58.6285650Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-04-20T20:13:58.6286523Z  # if we're on the base branch then use the parent commit 2024-04-20T20:13:58.6287229Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-04-20T20:13:58.6287746Z else 2024-04-20T20:13:58.6288315Z  # otherwise we're on a PR, so use the most recent base commit 2024-04-20T20:13:58.6289154Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-04-20T20:13:58.6289762Z fi 2024-04-20T20:13:58.6290077Z  2024-04-20T20:13:58.6290439Z if [[ -z "${MERGE_BASE}" ]]; then 2024-04-20T20:13:58.6291034Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6291583Z  2024-04-20T20:13:58.6292392Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-04-20T20:13:58.6293780Z  exit 0 2024-04-20T20:13:58.6294113Z fi 2024-04-20T20:13:58.6294433Z  2024-04-20T20:13:58.6294965Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-04-20T20:13:58.6296228Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-04-20T20:13:58.6297248Z  exit 1 2024-04-20T20:13:58.6297589Z fi 2024-04-20T20:13:58.6297909Z  2024-04-20T20:13:58.6298500Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-04-20T20:13:58.6299704Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-04-20T20:13:58.6300815Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-04-20T20:13:58.6302046Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-04-20T20:13:58.6303501Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-04-20T20:13:58.6304493Z fi 2024-04-20T20:13:58.6304817Z  2024-04-20T20:13:58.6305216Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-04-20T20:13:58.6317528Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:13:58.6318114Z env: 2024-04-20T20:13:58.6318442Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:13:58.6318870Z DOCKER_BUILD_DIR: .ci/docker 2024-04-20T20:13:58.6319415Z BASE_REVISION: f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:13:58.6321073Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6322683Z DOCKER_TAG: 47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:13:58.6323408Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:13:58.6324037Z ##[endgroup] 2024-04-20T20:13:58.6357317Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:13:58.6358141Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:13:58.6360167Z + aws ecr get-login-password --region us-east-1 2024-04-20T20:13:58.6361544Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:13:59.7639580Z WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json. 2024-04-20T20:13:59.7640621Z Configure a credential helper to remove this warning. See 2024-04-20T20:13:59.7641875Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-20T20:13:59.7643275Z 2024-04-20T20:13:59.7645477Z Login Succeeded 2024-04-20T20:13:59.7665210Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:00.4781153Z { 2024-04-20T20:14:00.4781572Z "schemaVersion": 2, 2024-04-20T20:14:00.4782510Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-04-20T20:14:00.4783271Z "config": { 2024-04-20T20:14:00.4783817Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-04-20T20:14:00.4784475Z "size": 45915, 2024-04-20T20:14:00.4785286Z "digest": "sha256:da1d6f3a8e8b44bd50c1c4b3f1a558025520fb11695fcf0b9b2c48ab31f6b21c" 2024-04-20T20:14:00.4786274Z }, 2024-04-20T20:14:00.4786576Z "layers": [ 2024-04-20T20:14:00.4786880Z { 2024-04-20T20:14:00.4787410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4788121Z "size": 28580681, 2024-04-20T20:14:00.4788961Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-04-20T20:14:00.4790305Z }, 2024-04-20T20:14:00.4790595Z { 2024-04-20T20:14:00.4791463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4792830Z "size": 7943451, 2024-04-20T20:14:00.4793897Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-04-20T20:14:00.4795135Z }, 2024-04-20T20:14:00.4795573Z { 2024-04-20T20:14:00.4796434Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4797536Z "size": 55728572, 2024-04-20T20:14:00.4798525Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-04-20T20:14:00.4799751Z }, 2024-04-20T20:14:00.4800265Z { 2024-04-20T20:14:00.4800936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4801698Z "size": 186, 2024-04-20T20:14:00.4802377Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-04-20T20:14:00.4803142Z }, 2024-04-20T20:14:00.4803424Z { 2024-04-20T20:14:00.4803945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4804628Z "size": 6886, 2024-04-20T20:14:00.4805285Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-04-20T20:14:00.4806047Z }, 2024-04-20T20:14:00.4806754Z { 2024-04-20T20:14:00.4807304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4807980Z "size": 1329001756, 2024-04-20T20:14:00.4808699Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-04-20T20:14:00.4809486Z }, 2024-04-20T20:14:00.4809765Z { 2024-04-20T20:14:00.4810274Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4811216Z "size": 62501, 2024-04-20T20:14:00.4811898Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-04-20T20:14:00.4812662Z }, 2024-04-20T20:14:00.4813149Z { 2024-04-20T20:14:00.4813716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4814404Z "size": 1684, 2024-04-20T20:14:00.4815060Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-04-20T20:14:00.4815794Z }, 2024-04-20T20:14:00.4816071Z { 2024-04-20T20:14:00.4816602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4817289Z "size": 1523, 2024-04-20T20:14:00.4817779Z + exit 0 2024-04-20T20:14:00.4818432Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-04-20T20:14:00.4819225Z }, 2024-04-20T20:14:00.4819497Z { 2024-04-20T20:14:00.4820019Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4820708Z "size": 2528295403, 2024-04-20T20:14:00.4821686Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-04-20T20:14:00.4822442Z }, 2024-04-20T20:14:00.4822729Z { 2024-04-20T20:14:00.4823262Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4823947Z "size": 86016, 2024-04-20T20:14:00.4824610Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-04-20T20:14:00.4825511Z }, 2024-04-20T20:14:00.4825799Z { 2024-04-20T20:14:00.4826315Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4826996Z "size": 1828, 2024-04-20T20:14:00.4827679Z "digest": "sha256:ad94e26724e095e0da03ae226814e01b2f4d01fdb657c92a9ab3c95e83d6b57f" 2024-04-20T20:14:00.4828450Z }, 2024-04-20T20:14:00.4828718Z { 2024-04-20T20:14:00.4829244Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4829925Z "size": 246728578, 2024-04-20T20:14:00.4830609Z "digest": "sha256:f3ab52783c7c2f73930c34fe153eb44413c0df28b629fe51265b31c81c6c22b2" 2024-04-20T20:14:00.4831397Z }, 2024-04-20T20:14:00.4831678Z { 2024-04-20T20:14:00.4832195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4832867Z "size": 544, 2024-04-20T20:14:00.4833556Z "digest": "sha256:2563aef2b5e0efdee23a0da27724abdea4bcbdec02ce7bb46b956d4044634229" 2024-04-20T20:14:00.4834351Z }, 2024-04-20T20:14:00.4834626Z { 2024-04-20T20:14:00.4835145Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4835828Z "size": 1282, 2024-04-20T20:14:00.4836483Z "digest": "sha256:02b6e595794c516df7978b78eaaba2803c327dd47141d449f028bea2acabf79f" 2024-04-20T20:14:00.4837240Z }, 2024-04-20T20:14:00.4837503Z { 2024-04-20T20:14:00.4838022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4838699Z "size": 484, 2024-04-20T20:14:00.4839383Z "digest": "sha256:b7be4cdf12beb6ccd661ebdf8c945e92c9164101f4779d55f8307edd3872ac9c" 2024-04-20T20:14:00.4840161Z }, 2024-04-20T20:14:00.4840439Z { 2024-04-20T20:14:00.4840956Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4841660Z "size": 91704877, 2024-04-20T20:14:00.4842319Z "digest": "sha256:36408387c0990ac22727194ab0e6a37ec2378b89cdd631f61192c4b142a002b9" 2024-04-20T20:14:00.4843064Z }, 2024-04-20T20:14:00.4843343Z { 2024-04-20T20:14:00.4843849Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4844688Z "size": 3263, 2024-04-20T20:14:00.4845352Z "digest": "sha256:72a46bc697fa0348f404e95db165b00fe627271fa30908fc8066424979ecc689" 2024-04-20T20:14:00.4846110Z }, 2024-04-20T20:14:00.4846375Z { 2024-04-20T20:14:00.4846893Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4847575Z "size": 1904, 2024-04-20T20:14:00.4848233Z "digest": "sha256:2d4080ccd1c38c3039faef3c0cfe8a1d6e8bb8675891df21b02bef452e20553f" 2024-04-20T20:14:00.4849025Z }, 2024-04-20T20:14:00.4849302Z { 2024-04-20T20:14:00.4849817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4850480Z "size": 700, 2024-04-20T20:14:00.4851173Z "digest": "sha256:c66e099c6afd28873bd7d291f7b6047093dde3f0992392054bd1a101a8db0b2d" 2024-04-20T20:14:00.4851935Z }, 2024-04-20T20:14:00.4852215Z { 2024-04-20T20:14:00.4852719Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4853671Z "size": 2774727940, 2024-04-20T20:14:00.4854374Z "digest": "sha256:afea1a7e623fafc75392a715ed8915719bf67562077c6723b219e532215ef3ed" 2024-04-20T20:14:00.4855144Z }, 2024-04-20T20:14:00.4855406Z { 2024-04-20T20:14:00.4855925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4856608Z "size": 380, 2024-04-20T20:14:00.4857241Z "digest": "sha256:80cce7440e991ad6596852c1f3d68765833e2f675e42eeb9214853eae6de8e66" 2024-04-20T20:14:00.4858000Z }, 2024-04-20T20:14:00.4858450Z { 2024-04-20T20:14:00.4858974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4859645Z "size": 11881, 2024-04-20T20:14:00.4860292Z "digest": "sha256:e8634591f94e0d6317c6c21685392889b62ec161a30fc072179e74c7390b5178" 2024-04-20T20:14:00.4861080Z }, 2024-04-20T20:14:00.4861362Z { 2024-04-20T20:14:00.4861869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4862558Z "size": 803, 2024-04-20T20:14:00.4863224Z "digest": "sha256:e939ec9a6e5a3b852e57414eb08145eff38b3839b87930a7232b4af9ec1b8f89" 2024-04-20T20:14:00.4863990Z }, 2024-04-20T20:14:00.4864257Z { 2024-04-20T20:14:00.4864777Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4865454Z "size": 106, 2024-04-20T20:14:00.4866094Z "digest": "sha256:f70b4789bdb56c36503a7f2fe597587694cf4ac4c60c3e3543052dc6cab921d2" 2024-04-20T20:14:00.4866854Z }, 2024-04-20T20:14:00.4867134Z { 2024-04-20T20:14:00.4867658Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4868321Z "size": 503, 2024-04-20T20:14:00.4868983Z "digest": "sha256:c2ff55aecf516b551480610594bf8e57fd4a5a2f3a56aaf347bc07b02cfc5a30" 2024-04-20T20:14:00.4869759Z }, 2024-04-20T20:14:00.4870039Z { 2024-04-20T20:14:00.4870546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4871263Z "size": 121477155, 2024-04-20T20:14:00.4871937Z "digest": "sha256:0683dc12345c674155473a1e0f177028e089738fd2e4df62cbc972056ead6d50" 2024-04-20T20:14:00.4872669Z }, 2024-04-20T20:14:00.4872953Z { 2024-04-20T20:14:00.4873472Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4874149Z "size": 109, 2024-04-20T20:14:00.4874793Z "digest": "sha256:213d8c928e3058efb2f13553670b76adfabfaf3f049677009dbc37dcafe59df4" 2024-04-20T20:14:00.4875554Z }, 2024-04-20T20:14:00.4875831Z { 2024-04-20T20:14:00.4876355Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4877026Z "size": 559, 2024-04-20T20:14:00.4877660Z "digest": "sha256:94f24c854d412b8e2d5646866c5572a476c187281c0f77aa44e14910201b3ea1" 2024-04-20T20:14:00.4878401Z }, 2024-04-20T20:14:00.4878665Z { 2024-04-20T20:14:00.4879183Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4879867Z "size": 1175621, 2024-04-20T20:14:00.4880540Z "digest": "sha256:eee943d09681b820705e97a5fce4b1310d5338d612f991beab1ee5332786da51" 2024-04-20T20:14:00.4881465Z }, 2024-04-20T20:14:00.4881758Z { 2024-04-20T20:14:00.4882283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4882973Z "size": 103, 2024-04-20T20:14:00.4883619Z "digest": "sha256:7d97e82bf7d2323784b5101745b74e9adfd76b3acd548f1e9c0924b478b9b056" 2024-04-20T20:14:00.4884378Z }, 2024-04-20T20:14:00.4884653Z { 2024-04-20T20:14:00.4885170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4885840Z "size": 1489, 2024-04-20T20:14:00.4886506Z "digest": "sha256:1e9dcf39486344df860f322cbc48f73323518f94d6af44aef52a4ed8356aa7e7" 2024-04-20T20:14:00.4887267Z }, 2024-04-20T20:14:00.4887532Z { 2024-04-20T20:14:00.4888057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4888738Z "size": 443477567, 2024-04-20T20:14:00.4889424Z "digest": "sha256:8f956acfb52b1b97c56066d0525e66c2099dc95f9cc24e694ce8a9a235ca1569" 2024-04-20T20:14:00.4890177Z }, 2024-04-20T20:14:00.4890456Z { 2024-04-20T20:14:00.4890975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4891690Z "size": 159, 2024-04-20T20:14:00.4892339Z "digest": "sha256:c5e368366dfd233e8cb330c9cc8838808bfbbc88ae6725470fda3488fcf4bec7" 2024-04-20T20:14:00.4893376Z }, 2024-04-20T20:14:00.4893663Z { 2024-04-20T20:14:00.4894170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4894856Z "size": 564, 2024-04-20T20:14:00.4895677Z "digest": "sha256:767e4ac29880dff2526038e4e22f07307731af72807c2480d9254ed705a5f1ae" 2024-04-20T20:14:00.4896423Z }, 2024-04-20T20:14:00.4896688Z { 2024-04-20T20:14:00.4897210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4897894Z "size": 33264432, 2024-04-20T20:14:00.4898583Z "digest": "sha256:620c62935ffaa7a5a5baa09d57c54126e4f4aeae97c941be7336480ef412300e" 2024-04-20T20:14:00.4899344Z }, 2024-04-20T20:14:00.4899624Z { 2024-04-20T20:14:00.4900148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4900829Z "size": 104, 2024-04-20T20:14:00.4901523Z "digest": "sha256:e9ebc6dc4418e373550f5108071e66cb82ade656a7f7422edf4ff5c30ebcc03f" 2024-04-20T20:14:00.4902292Z }, 2024-04-20T20:14:00.4902574Z { 2024-04-20T20:14:00.4903081Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4903754Z "size": 424, 2024-04-20T20:14:00.4904430Z "digest": "sha256:208b9e824b2eac3fe16c13faaef419b84d0351e4ed2fc4f3be4f3c766b15d6ce" 2024-04-20T20:14:00.4905219Z }, 2024-04-20T20:14:00.4905486Z { 2024-04-20T20:14:00.4906006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4906688Z "size": 20262057, 2024-04-20T20:14:00.4907360Z "digest": "sha256:d85f58908c4a4d5c69d719738f8d2d231620e5dc466bc3082eb4202a76c16de4" 2024-04-20T20:14:00.4908102Z }, 2024-04-20T20:14:00.4908381Z { 2024-04-20T20:14:00.4908902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4909571Z "size": 438, 2024-04-20T20:14:00.4910211Z "digest": "sha256:18747ce54e9e47c65013c12ffbc93be8e42356321970810a917158b03a0987af" 2024-04-20T20:14:00.4910960Z }, 2024-04-20T20:14:00.4911267Z { 2024-04-20T20:14:00.4911771Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4912454Z "size": 700, 2024-04-20T20:14:00.4913112Z "digest": "sha256:c66e099c6afd28873bd7d291f7b6047093dde3f0992392054bd1a101a8db0b2d" 2024-04-20T20:14:00.4913886Z }, 2024-04-20T20:14:00.4914149Z { 2024-04-20T20:14:00.4914668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4915348Z "size": 140, 2024-04-20T20:14:00.4916003Z "digest": "sha256:f77ca647b56304cc5d86473d0bbc33867ef59f984b6594e4d0fa452dce47ee2e" 2024-04-20T20:14:00.4916754Z }, 2024-04-20T20:14:00.4917031Z { 2024-04-20T20:14:00.4917572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4918394Z "size": 136, 2024-04-20T20:14:00.4919062Z "digest": "sha256:5ff7b5c83ad333ee8c168f7148c806b147847c23097172d1f7ba17e5e582d109" 2024-04-20T20:14:00.4919830Z }, 2024-04-20T20:14:00.4920115Z { 2024-04-20T20:14:00.4920624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4921352Z "size": 4865306376, 2024-04-20T20:14:00.4922036Z "digest": "sha256:317501565cc8c8c50cc07ee7e4478a5df4725febd68a3f34b1c630f329e4b4c8" 2024-04-20T20:14:00.4922804Z }, 2024-04-20T20:14:00.4923072Z { 2024-04-20T20:14:00.4923592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4924267Z "size": 190, 2024-04-20T20:14:00.4924933Z "digest": "sha256:0967a7f7b66dce95cc4c6add7d159917bc8648ea3018314931d8caeff065d925" 2024-04-20T20:14:00.4925694Z }, 2024-04-20T20:14:00.4925977Z { 2024-04-20T20:14:00.4926492Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4927175Z "size": 565, 2024-04-20T20:14:00.4927833Z "digest": "sha256:ad9c634231668b64d43855dc3134c69bf227de801be938384edf3cf39d51454b" 2024-04-20T20:14:00.4928588Z }, 2024-04-20T20:14:00.4928868Z { 2024-04-20T20:14:00.4929378Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4930058Z "size": 43165876, 2024-04-20T20:14:00.4930756Z "digest": "sha256:5eeedec4e70c9aa11df7b1bd9b981ac987db9919e099310971034a3c063367b0" 2024-04-20T20:14:00.4931568Z }, 2024-04-20T20:14:00.4931977Z { 2024-04-20T20:14:00.4932504Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4933437Z "size": 106, 2024-04-20T20:14:00.4934081Z "digest": "sha256:e42eb3b72f6296513464fc3e4efd0854d80c510e56519d3d42ba385a4092d4c7" 2024-04-20T20:14:00.4934845Z }, 2024-04-20T20:14:00.4935130Z { 2024-04-20T20:14:00.4935650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4936321Z "size": 1231, 2024-04-20T20:14:00.4937021Z "digest": "sha256:e6d2b60bdb9be5126d26ba68524c31dfc1cc53c0d9d4f7c185e45d1aa528f527" 2024-04-20T20:14:00.4937801Z }, 2024-04-20T20:14:00.4938078Z { 2024-04-20T20:14:00.4938592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4939309Z "size": 700, 2024-04-20T20:14:00.4939962Z "digest": "sha256:c66e099c6afd28873bd7d291f7b6047093dde3f0992392054bd1a101a8db0b2d" 2024-04-20T20:14:00.4940726Z }, 2024-04-20T20:14:00.4941032Z { 2024-04-20T20:14:00.4941566Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4942248Z "size": 138, 2024-04-20T20:14:00.4942896Z "digest": "sha256:7b9f9a9c789734e5010fc1d5ab205f975edc985b179f62491ccf9ee6f70aa152" 2024-04-20T20:14:00.4943666Z }, 2024-04-20T20:14:00.4943943Z { 2024-04-20T20:14:00.4944458Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4945125Z "size": 119, 2024-04-20T20:14:00.4945784Z "digest": "sha256:1d033bbde78f949a8da9325b5b3d7b95450d212b56757107b060b7bd4f98e7e8" 2024-04-20T20:14:00.4946543Z }, 2024-04-20T20:14:00.4946817Z { 2024-04-20T20:14:00.4947320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4948009Z "size": 1782603472, 2024-04-20T20:14:00.4948693Z "digest": "sha256:082d1697855caeea64859482b207baf31f6ef9ef44bc1efb068a9a5222969452" 2024-04-20T20:14:00.4949450Z }, 2024-04-20T20:14:00.4949719Z { 2024-04-20T20:14:00.4950240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4950923Z "size": 170, 2024-04-20T20:14:00.4951597Z "digest": "sha256:65973e7fcbaabb133382557df12e43a915edf2f07e662f80c8a56412209e36fb" 2024-04-20T20:14:00.4952368Z }, 2024-04-20T20:14:00.4952644Z { 2024-04-20T20:14:00.4953166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4953838Z "size": 1841, 2024-04-20T20:14:00.4954475Z "digest": "sha256:6597809d954c228121e3c388a670036b56d5f0052fc9d83c749053b705f8256c" 2024-04-20T20:14:00.4955405Z }, 2024-04-20T20:14:00.4955699Z { 2024-04-20T20:14:00.4956208Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4956896Z "size": 7529787, 2024-04-20T20:14:00.4957573Z "digest": "sha256:df9c106b278fec9b249345977082129d0d876315f96bff6ad7a93cd53fe6b067" 2024-04-20T20:14:00.4958320Z }, 2024-04-20T20:14:00.4958602Z { 2024-04-20T20:14:00.4959119Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4959804Z "size": 162, 2024-04-20T20:14:00.4960456Z "digest": "sha256:ac8a6e442af8ea2921d5e6d210aa87ca9a382092937a2fae83c9f3e91c6c4be5" 2024-04-20T20:14:00.4961264Z }, 2024-04-20T20:14:00.4961546Z { 2024-04-20T20:14:00.4962065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4962737Z "size": 7940, 2024-04-20T20:14:00.4963419Z "digest": "sha256:c4bbb701488afbf760dc533bfe4aeace99cd74146456f859622aa2eb386f3b0f" 2024-04-20T20:14:00.4964200Z }, 2024-04-20T20:14:00.4964470Z { 2024-04-20T20:14:00.4964992Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4965670Z "size": 8072, 2024-04-20T20:14:00.4966352Z "digest": "sha256:88e79a51b3abdf3a344422cd80ca3e54b3ff5bf4f5ea8eda4b2287fe98589c2f" 2024-04-20T20:14:00.4967122Z }, 2024-04-20T20:14:00.4967404Z { 2024-04-20T20:14:00.4967919Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4968596Z "size": 304, 2024-04-20T20:14:00.4969390Z "digest": "sha256:2b703eebf54c66272601662a0cd9a5af2ca4fb5a6d9d7c209ade0139900a32a9" 2024-04-20T20:14:00.4970164Z }, 2024-04-20T20:14:00.4970451Z { 2024-04-20T20:14:00.4970974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4971682Z "size": 7620527, 2024-04-20T20:14:00.4972357Z "digest": "sha256:e85e9d37ff5c71f05ea53b9e16111648f3adc18b9d19e75c9855e8921c93b3e9" 2024-04-20T20:14:00.4973374Z }, 2024-04-20T20:14:00.4973647Z { 2024-04-20T20:14:00.4974186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4974867Z "size": 108, 2024-04-20T20:14:00.4975536Z "digest": "sha256:5eff36389419c8ccb567ac5faac0a166f98dafec6548fd9e6b180a6b6bbffc05" 2024-04-20T20:14:00.4976305Z }, 2024-04-20T20:14:00.4995752Z { 2024-04-20T20:14:00.4996353Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.4997060Z "size": 54145479, 2024-04-20T20:14:00.4997777Z "digest": "sha256:e411beb65bc4e1d6acfee988010a6f592f6ae96bbcbb74cf496634a5bfe91b03" 2024-04-20T20:14:00.4998581Z }, 2024-04-20T20:14:00.4998858Z { 2024-04-20T20:14:00.4999387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5000073Z "size": 498, 2024-04-20T20:14:00.5000706Z "digest": "sha256:733067329e10f0c36e28e78ed630bce3c1c97198523a899b90ece9a4647376a0" 2024-04-20T20:14:00.5001516Z }, 2024-04-20T20:14:00.5001795Z { 2024-04-20T20:14:00.5002329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5003004Z "size": 1496846417, 2024-04-20T20:14:00.5003679Z "digest": "sha256:a2627550cf2f97ca918942a128195d732b43e0c1567dab85619c32df49a03305" 2024-04-20T20:14:00.5004427Z }, 2024-04-20T20:14:00.5004691Z { 2024-04-20T20:14:00.5005212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5005890Z "size": 106, 2024-04-20T20:14:00.5006562Z "digest": "sha256:6ad1f16aecb0d2275c19ddcc810818c363c69f9c577a6b8bc29a816e6e2c3a06" 2024-04-20T20:14:00.5007325Z }, 2024-04-20T20:14:00.5007601Z { 2024-04-20T20:14:00.5008120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5008800Z "size": 447, 2024-04-20T20:14:00.5009420Z "digest": "sha256:7d035680e047260b96536305413cdc734fa7caa35b646673283c106c516db010" 2024-04-20T20:14:00.5010164Z }, 2024-04-20T20:14:00.5010438Z { 2024-04-20T20:14:00.5010954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5011915Z "size": 46248376, 2024-04-20T20:14:00.5012600Z "digest": "sha256:64ac5416a9132c333802a5b38055d680a62dcd26aa5078154aed4c40286541cf" 2024-04-20T20:14:00.5013734Z }, 2024-04-20T20:14:00.5014009Z { 2024-04-20T20:14:00.5014529Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5015213Z "size": 111, 2024-04-20T20:14:00.5015893Z "digest": "sha256:bd67a4a30fb710b6eef41ab920b1622d8b611bea9fcb2224a0b7f18af66bf0b0" 2024-04-20T20:14:00.5016673Z }, 2024-04-20T20:14:00.5016949Z { 2024-04-20T20:14:00.5017481Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5018156Z "size": 32, 2024-04-20T20:14:00.5018804Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-20T20:14:00.5019584Z }, 2024-04-20T20:14:00.5019860Z { 2024-04-20T20:14:00.5020365Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5021091Z "size": 32, 2024-04-20T20:14:00.5021761Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-20T20:14:00.5022530Z }, 2024-04-20T20:14:00.5022797Z { 2024-04-20T20:14:00.5023309Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-04-20T20:14:00.5023991Z "size": 32, 2024-04-20T20:14:00.5024653Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-04-20T20:14:00.5025526Z } 2024-04-20T20:14:00.5025993Z ] 2024-04-20T20:14:00.5026270Z } 2024-04-20T20:14:00.5159822Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-04-20T20:14:00.5160414Z tag=${ECR_DOCKER_IMAGE##*/} 2024-04-20T20:14:00.5161151Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-04-20T20:14:00.5174916Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:14:00.5175528Z env: 2024-04-20T20:14:00.5175839Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:14:00.5177339Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:00.5178875Z ##[endgroup] 2024-04-20T20:14:00.5217935Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks-47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:00.5281454Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-04-20T20:14:00.5282154Z with: 2024-04-20T20:14:00.5283537Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:00.5285249Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:14:00.5285863Z env: 2024-04-20T20:14:00.5286176Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:14:00.5286579Z ##[endgroup] 2024-04-20T20:14:00.5306313Z ##[group]Run set -x 2024-04-20T20:14:00.5306718Z set -x 2024-04-20T20:14:00.5307067Z set +e 2024-04-20T20:14:00.5307408Z  2024-04-20T20:14:00.5307715Z login() { 2024-04-20T20:14:00.5308519Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-04-20T20:14:00.5309406Z } 2024-04-20T20:14:00.5309720Z  2024-04-20T20:14:00.5310081Z retry () { 2024-04-20T20:14:00.5310542Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-04-20T20:14:00.5311124Z } 2024-04-20T20:14:00.5311440Z  2024-04-20T20:14:00.5311805Z retry login "${DOCKER_REGISTRY}" 2024-04-20T20:14:00.5312286Z  2024-04-20T20:14:00.5312601Z set -e 2024-04-20T20:14:00.5313165Z # ignore output since only exit code is used for conditional 2024-04-20T20:14:00.5314022Z # only pull docker image if it's not available locally 2024-04-20T20:14:00.5314953Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-04-20T20:14:00.5315808Z  retry docker pull "${DOCKER_IMAGE}" 2024-04-20T20:14:00.5316337Z fi 2024-04-20T20:14:00.5328431Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:14:00.5329034Z env: 2024-04-20T20:14:00.5329352Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:14:00.5330819Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:00.5332571Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:14:00.5333454Z ##[endgroup] 2024-04-20T20:14:00.5365681Z + set +e 2024-04-20T20:14:00.5366570Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:14:00.5367388Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:14:00.5369934Z + aws ecr get-login-password --region us-east-1 2024-04-20T20:14:00.5371396Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-04-20T20:14:01.6673749Z WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json. 2024-04-20T20:14:01.6675133Z Configure a credential helper to remove this warning. See 2024-04-20T20:14:01.6676691Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-04-20T20:14:01.6677420Z 2024-04-20T20:14:01.6677571Z Login Succeeded 2024-04-20T20:14:01.6698764Z + set -e 2024-04-20T20:14:01.6700564Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:01.6911325Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:01.6914192Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:14:02.2427549Z 47c5092959bf58e951a46327442e23a676d2687b: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks 2024-04-20T20:14:02.2430097Z 7a2c55901189: Pulling fs layer 2024-04-20T20:14:02.2430814Z 224fe954d725: Pulling fs layer 2024-04-20T20:14:02.2432141Z 75722010b82e: Pulling fs layer 2024-04-20T20:14:02.2432962Z d527cbbb87e3: Pulling fs layer 2024-04-20T20:14:02.2433743Z b57676e46aee: Pulling fs layer 2024-04-20T20:14:02.2434599Z a8c1e85b5e14: Pulling fs layer 2024-04-20T20:14:02.2435442Z a41a8d1c11c8: Pulling fs layer 2024-04-20T20:14:02.2436196Z 0c1227890755: Pulling fs layer 2024-04-20T20:14:02.2436851Z d8d1234baab3: Pulling fs layer 2024-04-20T20:14:02.2437560Z 7ed32bc8e469: Pulling fs layer 2024-04-20T20:14:02.2438386Z ec1e7978c1fe: Pulling fs layer 2024-04-20T20:14:02.2439158Z ad94e26724e0: Pulling fs layer 2024-04-20T20:14:02.2439818Z f3ab52783c7c: Pulling fs layer 2024-04-20T20:14:02.2440531Z 2563aef2b5e0: Pulling fs layer 2024-04-20T20:14:02.2441422Z 02b6e595794c: Pulling fs layer 2024-04-20T20:14:02.2442269Z b7be4cdf12be: Pulling fs layer 2024-04-20T20:14:02.2443030Z 36408387c099: Pulling fs layer 2024-04-20T20:14:02.2443854Z 72a46bc697fa: Pulling fs layer 2024-04-20T20:14:02.2444650Z 2d4080ccd1c3: Pulling fs layer 2024-04-20T20:14:02.2445459Z c66e099c6afd: Pulling fs layer 2024-04-20T20:14:02.2446237Z ec1e7978c1fe: Waiting 2024-04-20T20:14:02.2446850Z afea1a7e623f: Pulling fs layer 2024-04-20T20:14:02.2447772Z b57676e46aee: Waiting 2024-04-20T20:14:02.2448503Z 80cce7440e99: Pulling fs layer 2024-04-20T20:14:02.2449278Z ad94e26724e0: Waiting 2024-04-20T20:14:02.2450009Z e8634591f94e: Pulling fs layer 2024-04-20T20:14:02.2450807Z a8c1e85b5e14: Waiting 2024-04-20T20:14:02.2451231Z f3ab52783c7c: Waiting 2024-04-20T20:14:02.2451606Z e939ec9a6e5a: Pulling fs layer 2024-04-20T20:14:02.2452033Z 02b6e595794c: Waiting 2024-04-20T20:14:02.2452416Z f70b4789bdb5: Pulling fs layer 2024-04-20T20:14:02.2453056Z 2563aef2b5e0: Waiting 2024-04-20T20:14:02.2453519Z c2ff55aecf51: Pulling fs layer 2024-04-20T20:14:02.2453962Z b7be4cdf12be: Waiting 2024-04-20T20:14:02.2454334Z a41a8d1c11c8: Waiting 2024-04-20T20:14:02.2454705Z 0683dc12345c: Pulling fs layer 2024-04-20T20:14:02.2455146Z 213d8c928e30: Pulling fs layer 2024-04-20T20:14:02.2455561Z 36408387c099: Waiting 2024-04-20T20:14:02.2455952Z 94f24c854d41: Pulling fs layer 2024-04-20T20:14:02.2456386Z eee943d09681: Pulling fs layer 2024-04-20T20:14:02.2456811Z d8d1234baab3: Waiting 2024-04-20T20:14:02.2457212Z 7d97e82bf7d2: Pulling fs layer 2024-04-20T20:14:02.2457625Z e939ec9a6e5a: Waiting 2024-04-20T20:14:02.2457994Z 80cce7440e99: Waiting 2024-04-20T20:14:02.2458380Z 1e9dcf394863: Pulling fs layer 2024-04-20T20:14:02.2458802Z e8634591f94e: Waiting 2024-04-20T20:14:02.2459158Z afea1a7e623f: Waiting 2024-04-20T20:14:02.2459546Z 8f956acfb52b: Pulling fs layer 2024-04-20T20:14:02.2459977Z d527cbbb87e3: Waiting 2024-04-20T20:14:02.2460343Z f70b4789bdb5: Waiting 2024-04-20T20:14:02.2460715Z c5e368366dfd: Pulling fs layer 2024-04-20T20:14:02.2461186Z 0683dc12345c: Waiting 2024-04-20T20:14:02.2461550Z 213d8c928e30: Waiting 2024-04-20T20:14:02.2461902Z c2ff55aecf51: Waiting 2024-04-20T20:14:02.2462268Z 94f24c854d41: Waiting 2024-04-20T20:14:02.2462645Z 767e4ac29880: Pulling fs layer 2024-04-20T20:14:02.2463081Z 7ed32bc8e469: Waiting 2024-04-20T20:14:02.2463437Z 7d97e82bf7d2: Waiting 2024-04-20T20:14:02.2463817Z 620c62935ffa: Pulling fs layer 2024-04-20T20:14:02.2464243Z 1e9dcf394863: Waiting 2024-04-20T20:14:02.2464903Z c5e368366dfd: Waiting 2024-04-20T20:14:02.2465272Z 8f956acfb52b: Waiting 2024-04-20T20:14:02.2465634Z 767e4ac29880: Waiting 2024-04-20T20:14:02.2466026Z e9ebc6dc4418: Pulling fs layer 2024-04-20T20:14:02.2466461Z 208b9e824b2e: Pulling fs layer 2024-04-20T20:14:02.2466904Z d85f58908c4a: Pulling fs layer 2024-04-20T20:14:02.2467330Z 72a46bc697fa: Waiting 2024-04-20T20:14:02.2467715Z 18747ce54e9e: Pulling fs layer 2024-04-20T20:14:02.2468126Z 208b9e824b2e: Waiting 2024-04-20T20:14:02.2468490Z d85f58908c4a: Waiting 2024-04-20T20:14:02.2468878Z f77ca647b563: Pulling fs layer 2024-04-20T20:14:02.2469284Z 0c1227890755: Waiting 2024-04-20T20:14:02.2469668Z 5ff7b5c83ad3: Pulling fs layer 2024-04-20T20:14:02.2470106Z 317501565cc8: Pulling fs layer 2024-04-20T20:14:02.2470529Z 18747ce54e9e: Waiting 2024-04-20T20:14:02.2471091Z 0967a7f7b66d: Pulling fs layer 2024-04-20T20:14:02.2471520Z f77ca647b563: Waiting 2024-04-20T20:14:02.2471902Z ad9c63423166: Pulling fs layer 2024-04-20T20:14:02.2472331Z 317501565cc8: Waiting 2024-04-20T20:14:02.2472681Z 5ff7b5c83ad3: Waiting 2024-04-20T20:14:02.2473078Z 5eeedec4e70c: Pulling fs layer 2024-04-20T20:14:02.2473526Z e42eb3b72f62: Pulling fs layer 2024-04-20T20:14:02.2473956Z e6d2b60bdb9b: Pulling fs layer 2024-04-20T20:14:02.2474403Z 7b9f9a9c7897: Pulling fs layer 2024-04-20T20:14:02.2474845Z 1d033bbde78f: Pulling fs layer 2024-04-20T20:14:02.2475288Z 082d1697855c: Pulling fs layer 2024-04-20T20:14:02.2475692Z e42eb3b72f62: Waiting 2024-04-20T20:14:02.2476065Z e6d2b60bdb9b: Waiting 2024-04-20T20:14:02.2476437Z 5eeedec4e70c: Waiting 2024-04-20T20:14:02.2476829Z 65973e7fcbaa: Pulling fs layer 2024-04-20T20:14:02.2477253Z 6597809d954c: Pulling fs layer 2024-04-20T20:14:02.2477691Z df9c106b278f: Pulling fs layer 2024-04-20T20:14:02.2478112Z 6597809d954c: Waiting 2024-04-20T20:14:02.2478485Z ac8a6e442af8: Pulling fs layer 2024-04-20T20:14:02.2478923Z c4bbb701488a: Pulling fs layer 2024-04-20T20:14:02.2479344Z 7b9f9a9c7897: Waiting 2024-04-20T20:14:02.2479734Z 88e79a51b3ab: Pulling fs layer 2024-04-20T20:14:02.2480163Z 2b703eebf54c: Pulling fs layer 2024-04-20T20:14:02.2480606Z e85e9d37ff5c: Pulling fs layer 2024-04-20T20:14:02.2481085Z 5eff36389419: Pulling fs layer 2024-04-20T20:14:02.2481488Z ac8a6e442af8: Waiting 2024-04-20T20:14:02.2481879Z e411beb65bc4: Pulling fs layer 2024-04-20T20:14:02.2482304Z 1d033bbde78f: Waiting 2024-04-20T20:14:02.2482670Z 88e79a51b3ab: Waiting 2024-04-20T20:14:02.2483040Z 733067329e10: Pulling fs layer 2024-04-20T20:14:02.2483453Z e85e9d37ff5c: Waiting 2024-04-20T20:14:02.2483839Z a2627550cf2f: Pulling fs layer 2024-04-20T20:14:02.2484259Z e411beb65bc4: Waiting 2024-04-20T20:14:02.2484635Z 6ad1f16aecb0: Pulling fs layer 2024-04-20T20:14:02.2485074Z 7d035680e047: Pulling fs layer 2024-04-20T20:14:02.2485503Z 64ac5416a913: Pulling fs layer 2024-04-20T20:14:02.2485935Z bd67a4a30fb7: Pulling fs layer 2024-04-20T20:14:02.2486355Z 082d1697855c: Waiting 2024-04-20T20:14:02.2486737Z 4f4fb700ef54: Pulling fs layer 2024-04-20T20:14:02.2487165Z 6ad1f16aecb0: Waiting 2024-04-20T20:14:02.2487516Z 7d035680e047: Waiting 2024-04-20T20:14:02.2487875Z 4f4fb700ef54: Waiting 2024-04-20T20:14:02.2488244Z 64ac5416a913: Waiting 2024-04-20T20:14:02.2488607Z 2b703eebf54c: Waiting 2024-04-20T20:14:02.2488958Z a2627550cf2f: Waiting 2024-04-20T20:14:02.2489320Z bd67a4a30fb7: Waiting 2024-04-20T20:14:02.2489686Z 5eff36389419: Waiting 2024-04-20T20:14:02.7885182Z 224fe954d725: Verifying Checksum 2024-04-20T20:14:02.7885775Z 224fe954d725: Download complete 2024-04-20T20:14:03.0432241Z 7a2c55901189: Verifying Checksum 2024-04-20T20:14:03.0432815Z 7a2c55901189: Download complete 2024-04-20T20:14:03.0590793Z d527cbbb87e3: Verifying Checksum 2024-04-20T20:14:03.0591352Z d527cbbb87e3: Download complete 2024-04-20T20:14:03.3095112Z b57676e46aee: Download complete 2024-04-20T20:14:03.6441180Z a41a8d1c11c8: Verifying Checksum 2024-04-20T20:14:03.6442008Z a41a8d1c11c8: Download complete 2024-04-20T20:14:03.7643934Z 7a2c55901189: Pull complete 2024-04-20T20:14:03.8324004Z 75722010b82e: Verifying Checksum 2024-04-20T20:14:03.8324921Z 75722010b82e: Download complete 2024-04-20T20:14:03.9247589Z 0c1227890755: Verifying Checksum 2024-04-20T20:14:03.9248538Z 0c1227890755: Download complete 2024-04-20T20:14:03.9873570Z 224fe954d725: Pull complete 2024-04-20T20:14:04.0981109Z d8d1234baab3: Verifying Checksum 2024-04-20T20:14:04.0981986Z d8d1234baab3: Download complete 2024-04-20T20:14:04.4384740Z ec1e7978c1fe: Verifying Checksum 2024-04-20T20:14:04.4385687Z ec1e7978c1fe: Download complete 2024-04-20T20:14:04.7055872Z ad94e26724e0: Verifying Checksum 2024-04-20T20:14:04.7056610Z ad94e26724e0: Download complete 2024-04-20T20:14:05.0313859Z 75722010b82e: Pull complete 2024-04-20T20:14:05.0874742Z d527cbbb87e3: Pull complete 2024-04-20T20:14:05.1434612Z b57676e46aee: Pull complete 2024-04-20T20:14:10.1374531Z f3ab52783c7c: Verifying Checksum 2024-04-20T20:14:10.1375908Z f3ab52783c7c: Download complete 2024-04-20T20:14:10.4365197Z 2563aef2b5e0: Verifying Checksum 2024-04-20T20:14:10.4366164Z 2563aef2b5e0: Download complete 2024-04-20T20:14:10.7136746Z 02b6e595794c: Download complete 2024-04-20T20:14:11.0436037Z b7be4cdf12be: Verifying Checksum 2024-04-20T20:14:11.0436917Z b7be4cdf12be: Download complete 2024-04-20T20:14:13.3109691Z 36408387c099: Verifying Checksum 2024-04-20T20:14:13.3110237Z 36408387c099: Download complete 2024-04-20T20:14:13.6006804Z 72a46bc697fa: Download complete 2024-04-20T20:14:13.8796696Z 2d4080ccd1c3: Download complete 2024-04-20T20:14:14.1545635Z c66e099c6afd: Download complete 2024-04-20T20:14:16.8444539Z a8c1e85b5e14: Download complete 2024-04-20T20:14:17.1488180Z 80cce7440e99: Download complete 2024-04-20T20:14:17.4202897Z e8634591f94e: Download complete 2024-04-20T20:14:17.7075620Z e939ec9a6e5a: Verifying Checksum 2024-04-20T20:14:17.7076587Z e939ec9a6e5a: Download complete 2024-04-20T20:14:17.9978289Z f70b4789bdb5: Verifying Checksum 2024-04-20T20:14:17.9979151Z f70b4789bdb5: Download complete 2024-04-20T20:14:18.2772193Z c2ff55aecf51: Verifying Checksum 2024-04-20T20:14:18.2773198Z c2ff55aecf51: Download complete 2024-04-20T20:14:20.7683350Z 0683dc12345c: Download complete 2024-04-20T20:14:21.0520415Z 213d8c928e30: Verifying Checksum 2024-04-20T20:14:21.0521071Z 213d8c928e30: Download complete 2024-04-20T20:14:21.3393796Z 94f24c854d41: Verifying Checksum 2024-04-20T20:14:21.3394673Z 94f24c854d41: Download complete 2024-04-20T20:14:21.8124438Z eee943d09681: Verifying Checksum 2024-04-20T20:14:21.8125471Z eee943d09681: Download complete 2024-04-20T20:14:22.1174690Z 7d97e82bf7d2: Verifying Checksum 2024-04-20T20:14:22.1175413Z 7d97e82bf7d2: Download complete 2024-04-20T20:14:22.4545309Z 1e9dcf394863: Verifying Checksum 2024-04-20T20:14:29.5055616Z 7ed32bc8e469: Verifying Checksum 2024-04-20T20:14:29.5056648Z 7ed32bc8e469: Download complete 2024-04-20T20:14:29.8246442Z c5e368366dfd: Verifying Checksum 2024-04-20T20:14:29.8247221Z c5e368366dfd: Download complete 2024-04-20T20:14:29.9483516Z 8f956acfb52b: Verifying Checksum 2024-04-20T20:14:29.9484098Z 8f956acfb52b: Download complete 2024-04-20T20:14:30.1358285Z 767e4ac29880: Verifying Checksum 2024-04-20T20:14:30.1358866Z 767e4ac29880: Download complete 2024-04-20T20:14:30.4941813Z e9ebc6dc4418: Verifying Checksum 2024-04-20T20:14:30.4942411Z e9ebc6dc4418: Download complete 2024-04-20T20:14:30.8310823Z 208b9e824b2e: Download complete 2024-04-20T20:14:30.9600911Z 620c62935ffa: Verifying Checksum 2024-04-20T20:14:30.9601557Z 620c62935ffa: Download complete 2024-04-20T20:14:31.2736005Z 18747ce54e9e: Verifying Checksum 2024-04-20T20:14:31.6056475Z f77ca647b563: Verifying Checksum 2024-04-20T20:14:31.6057310Z f77ca647b563: Download complete 2024-04-20T20:14:31.6715473Z d85f58908c4a: Verifying Checksum 2024-04-20T20:14:31.6716372Z d85f58908c4a: Download complete 2024-04-20T20:14:31.9222742Z 5ff7b5c83ad3: Download complete 2024-04-20T20:14:32.2489136Z 0967a7f7b66d: Verifying Checksum 2024-04-20T20:14:32.2489901Z 0967a7f7b66d: Download complete 2024-04-20T20:14:32.3541589Z a8c1e85b5e14: Pull complete 2024-04-20T20:14:32.4179545Z a41a8d1c11c8: Pull complete 2024-04-20T20:14:32.4788044Z 0c1227890755: Pull complete 2024-04-20T20:14:32.5348473Z d8d1234baab3: Pull complete 2024-04-20T20:14:32.5709588Z ad9c63423166: Download complete 2024-04-20T20:14:33.6992008Z 5eeedec4e70c: Verifying Checksum 2024-04-20T20:14:33.6993003Z 5eeedec4e70c: Download complete 2024-04-20T20:14:34.0303727Z e42eb3b72f62: Download complete 2024-04-20T20:14:34.3505081Z e6d2b60bdb9b: Verifying Checksum 2024-04-20T20:14:34.3505660Z e6d2b60bdb9b: Download complete 2024-04-20T20:14:34.6443743Z 7b9f9a9c7897: Download complete 2024-04-20T20:14:34.9803603Z 1d033bbde78f: Verifying Checksum 2024-04-20T20:14:34.9804181Z 1d033bbde78f: Download complete 2024-04-20T20:15:00.9141048Z afea1a7e623f: Verifying Checksum 2024-04-20T20:15:00.9141687Z afea1a7e623f: Download complete 2024-04-20T20:15:01.2779639Z 65973e7fcbaa: Verifying Checksum 2024-04-20T20:15:01.2780223Z 65973e7fcbaa: Download complete 2024-04-20T20:15:01.6004977Z 6597809d954c: Download complete 2024-04-20T20:15:02.3045621Z df9c106b278f: Download complete 2024-04-20T20:15:02.6798094Z ac8a6e442af8: Download complete 2024-04-20T20:15:03.0105120Z c4bbb701488a: Download complete 2024-04-20T20:15:03.3443050Z 88e79a51b3ab: Verifying Checksum 2024-04-20T20:15:03.3443912Z 88e79a51b3ab: Download complete 2024-04-20T20:15:03.6664480Z 2b703eebf54c: Download complete 2024-04-20T20:15:04.3420958Z e85e9d37ff5c: Verifying Checksum 2024-04-20T20:15:04.3422660Z e85e9d37ff5c: Download complete 2024-04-20T20:15:04.3567156Z 082d1697855c: Verifying Checksum 2024-04-20T20:15:04.3568070Z 082d1697855c: Download complete 2024-04-20T20:15:04.6704457Z 5eff36389419: Download complete 2024-04-20T20:15:04.9725200Z 733067329e10: Verifying Checksum 2024-04-20T20:15:04.9726040Z 733067329e10: Download complete 2024-04-20T20:15:06.1616032Z e411beb65bc4: Verifying Checksum 2024-04-20T20:15:06.1616892Z e411beb65bc4: Download complete 2024-04-20T20:15:06.3237851Z 7ed32bc8e469: Pull complete 2024-04-20T20:15:06.4653396Z 6ad1f16aecb0: Verifying Checksum 2024-04-20T20:15:06.4654473Z 6ad1f16aecb0: Download complete 2024-04-20T20:15:06.7184628Z ec1e7978c1fe: Pull complete 2024-04-20T20:15:06.8324133Z 7d035680e047: Verifying Checksum 2024-04-20T20:15:06.8324908Z 7d035680e047: Download complete 2024-04-20T20:15:07.1315868Z ad94e26724e0: Pull complete 2024-04-20T20:15:08.1013817Z 64ac5416a913: Verifying Checksum 2024-04-20T20:15:08.1014409Z 64ac5416a913: Download complete 2024-04-20T20:15:08.4431450Z bd67a4a30fb7: Verifying Checksum 2024-04-20T20:15:08.4432289Z bd67a4a30fb7: Download complete 2024-04-20T20:15:08.5683079Z 4f4fb700ef54: Verifying Checksum 2024-04-20T20:15:08.5683648Z 4f4fb700ef54: Download complete 2024-04-20T20:15:13.4699410Z f3ab52783c7c: Pull complete 2024-04-20T20:15:14.0071201Z 2563aef2b5e0: Pull complete 2024-04-20T20:15:14.3744725Z 02b6e595794c: Pull complete 2024-04-20T20:15:14.7596854Z b7be4cdf12be: Pull complete 2024-04-20T20:15:17.2607831Z 36408387c099: Pull complete 2024-04-20T20:15:17.7500892Z 72a46bc697fa: Pull complete 2024-04-20T20:15:18.1034376Z 2d4080ccd1c3: Pull complete 2024-04-20T20:15:18.3186371Z c66e099c6afd: Pull complete 2024-04-20T20:15:31.1427371Z a2627550cf2f: Download complete 2024-04-20T20:15:53.3616032Z 317501565cc8: Verifying Checksum 2024-04-20T20:15:53.3616958Z 317501565cc8: Download complete 2024-04-20T20:16:02.1105403Z afea1a7e623f: Pull complete 2024-04-20T20:16:02.1675971Z 80cce7440e99: Pull complete 2024-04-20T20:16:02.2203498Z e8634591f94e: Pull complete 2024-04-20T20:16:02.2698639Z e939ec9a6e5a: Pull complete 2024-04-20T20:16:02.3198116Z f70b4789bdb5: Pull complete 2024-04-20T20:16:02.3678637Z c2ff55aecf51: Pull complete 2024-04-20T20:16:06.1615811Z 0683dc12345c: Pull complete 2024-04-20T20:16:06.6141616Z 213d8c928e30: Pull complete 2024-04-20T20:16:06.8693671Z 94f24c854d41: Pull complete 2024-04-20T20:16:07.1253223Z eee943d09681: Pull complete 2024-04-20T20:16:07.2721088Z 7d97e82bf7d2: Pull complete 2024-04-20T20:16:07.3221990Z 1e9dcf394863: Pull complete 2024-04-20T20:16:14.1891535Z 8f956acfb52b: Pull complete 2024-04-20T20:16:14.2652704Z c5e368366dfd: Pull complete 2024-04-20T20:16:14.4523464Z 767e4ac29880: Pull complete 2024-04-20T20:16:15.3549733Z 620c62935ffa: Pull complete 2024-04-20T20:16:15.4059599Z e9ebc6dc4418: Pull complete 2024-04-20T20:16:15.4553315Z 208b9e824b2e: Pull complete 2024-04-20T20:16:15.7771017Z d85f58908c4a: Pull complete 2024-04-20T20:16:15.8260956Z 18747ce54e9e: Pull complete 2024-04-20T20:16:15.9256197Z f77ca647b563: Pull complete 2024-04-20T20:16:15.9743831Z 5ff7b5c83ad3: Pull complete 2024-04-20T20:16:58.8557091Z 317501565cc8: Pull complete 2024-04-20T20:16:58.9072734Z 0967a7f7b66d: Pull complete 2024-04-20T20:16:58.9579059Z ad9c63423166: Pull complete 2024-04-20T20:17:00.1189829Z 5eeedec4e70c: Pull complete 2024-04-20T20:17:00.1681001Z e42eb3b72f62: Pull complete 2024-04-20T20:17:00.2185062Z e6d2b60bdb9b: Pull complete 2024-04-20T20:17:00.3171953Z 7b9f9a9c7897: Pull complete 2024-04-20T20:17:00.3658246Z 1d033bbde78f: Pull complete 2024-04-20T20:17:33.3755728Z 082d1697855c: Pull complete 2024-04-20T20:17:33.8331637Z 65973e7fcbaa: Pull complete 2024-04-20T20:17:34.3589437Z 6597809d954c: Pull complete 2024-04-20T20:17:34.8262096Z df9c106b278f: Pull complete 2024-04-20T20:17:34.8787920Z ac8a6e442af8: Pull complete 2024-04-20T20:17:34.9286982Z c4bbb701488a: Pull complete 2024-04-20T20:17:34.9783156Z 88e79a51b3ab: Pull complete 2024-04-20T20:17:35.0264465Z 2b703eebf54c: Pull complete 2024-04-20T20:17:35.4698004Z e85e9d37ff5c: Pull complete 2024-04-20T20:17:35.5190627Z 5eff36389419: Pull complete 2024-04-20T20:17:37.3139678Z e411beb65bc4: Pull complete 2024-04-20T20:17:37.3632979Z 733067329e10: Pull complete 2024-04-20T20:17:54.7609334Z a2627550cf2f: Pull complete 2024-04-20T20:17:54.8154390Z 6ad1f16aecb0: Pull complete 2024-04-20T20:17:54.8661489Z 7d035680e047: Pull complete 2024-04-20T20:17:55.6666005Z 64ac5416a913: Pull complete 2024-04-20T20:17:55.7166431Z bd67a4a30fb7: Pull complete 2024-04-20T20:17:55.7639594Z 4f4fb700ef54: Pull complete 2024-04-20T20:18:06.8740579Z Digest: sha256:d2743ea4ce53dd66cc45d04e9ebd8705c0801917130f6a1470ec14eb256d9123 2024-04-20T20:18:06.8770538Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:18:06.8793422Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:18:06.8922866Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-04-20T20:18:06.8924033Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-04-20T20:18:06.8939354Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:06.8939967Z env: 2024-04-20T20:18:06.8940290Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:06.8940748Z ##[endgroup] 2024-04-20T20:18:06.9208101Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-04-20T20:18:06.9208797Z with: 2024-04-20T20:18:06.9209127Z driver-version: 535.54.03 2024-04-20T20:18:06.9209524Z env: 2024-04-20T20:18:06.9209824Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:06.9210230Z ##[endgroup] 2024-04-20T20:18:06.9264922Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-20T20:18:06.9265676Z with: 2024-04-20T20:18:06.9265992Z timeout_minutes: 10 2024-04-20T20:18:06.9266377Z max_attempts: 3 2024-04-20T20:18:06.9308520Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-04-20T20:18:06.9350986Z retry_wait_seconds: 10 2024-04-20T20:18:06.9351421Z polling_interval_seconds: 1 2024-04-20T20:18:06.9351852Z warning_on_retry: true 2024-04-20T20:18:06.9352265Z continue_on_error: false 2024-04-20T20:18:06.9352662Z env: 2024-04-20T20:18:06.9352972Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:06.9353375Z DRIVER_VERSION: 535.54.03 2024-04-20T20:18:06.9353782Z ##[endgroup] 2024-04-20T20:18:07.0182201Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run == 2024-04-20T20:18:07.0184139Z + install_nvidia_driver_common 2024-04-20T20:18:07.0186442Z + echo 'Before installing NVIDIA driver' 2024-04-20T20:18:07.0189716Z Before installing NVIDIA driver 2024-04-20T20:18:07.0190805Z + lspci 2024-04-20T20:18:07.0303386Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-20T20:18:07.0305036Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03) 2024-04-20T20:18:07.0306184Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 2024-04-20T20:18:07.0307181Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI 2024-04-20T20:18:07.0307989Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1) 2024-04-20T20:18:07.0308821Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device 2024-04-20T20:18:07.0309613Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG 2024-04-20T20:18:07.0310219Z + lsmod 2024-04-20T20:18:07.0326853Z Module Size Used by 2024-04-20T20:18:07.0327669Z ib_core 393216 0 2024-04-20T20:18:07.0328458Z nvidia_modeset 1302528 0 2024-04-20T20:18:07.0329216Z veth 32768 0 2024-04-20T20:18:07.0329937Z nvidia_uvm 1540096 0 2024-04-20T20:18:07.0330585Z nvidia 56455168 2 nvidia_uvm,nvidia_modeset 2024-04-20T20:18:07.0331511Z drm 618496 1 nvidia 2024-04-20T20:18:07.0332188Z xt_conntrack 16384 1 2024-04-20T20:18:07.0332642Z xt_MASQUERADE 20480 1 2024-04-20T20:18:07.0333511Z xfrm_user 40960 1 2024-04-20T20:18:07.0333991Z xfrm_algo 16384 1 xfrm_user 2024-04-20T20:18:07.0334496Z xt_addrtype 16384 2 2024-04-20T20:18:07.0334946Z iptable_filter 16384 1 2024-04-20T20:18:07.0335396Z iptable_nat 16384 1 2024-04-20T20:18:07.0335914Z nf_nat 49152 2 iptable_nat,xt_MASQUERADE 2024-04-20T20:18:07.0336490Z bpfilter 16384 0 2024-04-20T20:18:07.0336932Z br_netfilter 28672 0 2024-04-20T20:18:07.0337733Z bridge 307200 1 br_netfilter 2024-04-20T20:18:07.0338274Z stp 16384 1 bridge 2024-04-20T20:18:07.0338785Z llc 16384 2 bridge,stp 2024-04-20T20:18:07.0339274Z aufs 270336 0 2024-04-20T20:18:07.0339686Z overlay 151552 0 2024-04-20T20:18:07.0340122Z nls_iso8859_1 16384 1 2024-04-20T20:18:07.0340559Z dm_multipath 40960 0 2024-04-20T20:18:07.0340992Z scsi_dh_rdac 16384 0 2024-04-20T20:18:07.0341548Z scsi_dh_emc 16384 0 2024-04-20T20:18:07.0342225Z scsi_dh_alua 20480 0 2024-04-20T20:18:07.0342929Z binfmt_misc 24576 1 2024-04-20T20:18:07.0343650Z crct10dif_pclmul 16384 1 2024-04-20T20:18:07.0344389Z crc32_pclmul 16384 0 2024-04-20T20:18:07.0345149Z virtio_net 61440 0 2024-04-20T20:18:07.0345902Z ghash_clmulni_intel 16384 0 2024-04-20T20:18:07.0346488Z net_failover 20480 1 virtio_net 2024-04-20T20:18:07.0347022Z aesni_intel 376832 0 2024-04-20T20:18:07.0347465Z psmouse 176128 0 2024-04-20T20:18:07.0348423Z input_leds 16384 0 2024-04-20T20:18:07.0349205Z failover 16384 1 net_failover 2024-04-20T20:18:07.0350106Z crypto_simd 16384 1 aesni_intel 2024-04-20T20:18:07.0350842Z cryptd 24576 2 crypto_simd,ghash_clmulni_intel 2024-04-20T20:18:07.0351843Z serio_raw 20480 0 2024-04-20T20:18:07.0352590Z sch_fq_codel 24576 13 2024-04-20T20:18:07.0353064Z efi_pstore 16384 0 2024-04-20T20:18:07.0353507Z virtio_rng 16384 0 2024-04-20T20:18:07.0354058Z ip_tables 32768 2 iptable_filter,iptable_nat 2024-04-20T20:18:07.0355176Z x_tables 53248 6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE 2024-04-20T20:18:07.0356619Z autofs4 49152 2 2024-04-20T20:18:07.0357336Z + modinfo nvidia 2024-04-20T20:18:07.0358446Z filename: /lib/modules/5.15.0-1041-gcp/kernel/drivers/video/nvidia.ko 2024-04-20T20:18:07.0359203Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-20T20:18:07.0359800Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-20T20:18:07.0360399Z alias: char-major-195-* 2024-04-20T20:18:07.0360847Z version: 535.54.03 2024-04-20T20:18:07.0361300Z supported: external 2024-04-20T20:18:07.0361712Z license: NVIDIA 2024-04-20T20:18:07.0362135Z srcversion: 415DC1001E7D8AD79B6207C 2024-04-20T20:18:07.0362704Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-20T20:18:07.0363298Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-20T20:18:07.0363893Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-20T20:18:07.0364417Z depends: drm 2024-04-20T20:18:07.0364768Z retpoline: Y 2024-04-20T20:18:07.0365118Z name: nvidia 2024-04-20T20:18:07.0365716Z vermagic: 5.15.0-1041-gcp SMP mod_unload modversions 2024-04-20T20:18:07.0366447Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-20T20:18:07.0367233Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-20T20:18:07.0367977Z parm: NVreg_ResmanDebugLevel:int 2024-04-20T20:18:07.0368509Z parm: NVreg_RmLogonRC:int 2024-04-20T20:18:07.0369025Z parm: NVreg_ModifyDeviceFiles:int 2024-04-20T20:18:07.0369559Z parm: NVreg_DeviceFileUID:int 2024-04-20T20:18:07.0370082Z parm: NVreg_DeviceFileGID:int 2024-04-20T20:18:07.0370604Z parm: NVreg_DeviceFileMode:int 2024-04-20T20:18:07.0371288Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-20T20:18:07.0371971Z parm: NVreg_UsePageAttributeTable:int 2024-04-20T20:18:07.0372549Z parm: NVreg_EnablePCIeGen3:int 2024-04-20T20:18:07.0373347Z parm: NVreg_EnableMSI:int 2024-04-20T20:18:07.0373840Z parm: NVreg_TCEBypassMode:int 2024-04-20T20:18:07.0374394Z parm: NVreg_EnableStreamMemOPs:int 2024-04-20T20:18:07.0375308Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-20T20:18:07.0376040Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-20T20:18:07.0376736Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-20T20:18:07.0377491Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-20T20:18:07.0378231Z parm: NVreg_DynamicPowerManagement:int 2024-04-20T20:18:07.0378984Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-20T20:18:07.0379706Z parm: NVreg_EnableGpuFirmware:int 2024-04-20T20:18:07.0380300Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-20T20:18:07.0380951Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-20T20:18:07.0381673Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-20T20:18:07.0382260Z parm: NVreg_MemoryPoolSize:int 2024-04-20T20:18:07.0382818Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-20T20:18:07.0383401Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-20T20:18:07.0383975Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-20T20:18:07.0384646Z parm: NVreg_NvLinkDisable:int 2024-04-20T20:18:07.0385267Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-20T20:18:07.0385916Z parm: NVreg_RegisterPCIDriver:int 2024-04-20T20:18:07.0386491Z parm: NVreg_EnableResizableBar:int 2024-04-20T20:18:07.0387065Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-20T20:18:07.0387647Z parm: NVreg_RegistryDwords:charp 2024-04-20T20:18:07.0388249Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-20T20:18:07.0388828Z parm: NVreg_RmMsg:charp 2024-04-20T20:18:07.0389305Z parm: NVreg_GpuBlacklist:charp 2024-04-20T20:18:07.0389863Z parm: NVreg_TemporaryFilePath:charp 2024-04-20T20:18:07.0390430Z parm: NVreg_ExcludedGpus:charp 2024-04-20T20:18:07.0390962Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-20T20:18:07.0391573Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-20T20:18:07.0392139Z parm: rm_firmware_active:charp 2024-04-20T20:18:07.0392634Z + HAS_NVIDIA_DRIVER=0 2024-04-20T20:18:07.0393115Z ++ command -v nvidia-smi 2024-04-20T20:18:07.0393601Z + '[' -x /usr/bin/nvidia-smi ']' 2024-04-20T20:18:07.0394031Z + set +e 2024-04-20T20:18:07.0394651Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-04-20T20:18:07.0566794Z + INSTALLED_DRIVER_VERSION=535.54.03 2024-04-20T20:18:07.0567834Z + NVIDIA_SMI_STATUS=0 2024-04-20T20:18:07.0568419Z + '[' 0 -ne 0 ']' 2024-04-20T20:18:07.0568843Z + '[' 535.54.03 '!=' 535.54.03 ']' 2024-04-20T20:18:07.0569298Z + HAS_NVIDIA_DRIVER=1 2024-04-20T20:18:07.0570204Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation' 2024-04-20T20:18:07.0571078Z + set -e 2024-04-20T20:18:07.0571501Z + '[' 1 -eq 0 ']' 2024-04-20T20:18:07.0572215Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation 2024-04-20T20:18:07.0573390Z + post_install_nvidia_driver_common 2024-04-20T20:18:07.0573876Z + sudo modprobe nvidia 2024-04-20T20:18:07.0714380Z + echo 'After installing NVIDIA driver' 2024-04-20T20:18:07.0715130Z + lspci 2024-04-20T20:18:07.0715487Z After installing NVIDIA driver 2024-04-20T20:18:07.0824443Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-04-20T20:18:07.0825940Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03) 2024-04-20T20:18:07.0826835Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 2024-04-20T20:18:07.0827790Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI 2024-04-20T20:18:07.0828587Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1) 2024-04-20T20:18:07.0829410Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device 2024-04-20T20:18:07.0830221Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG 2024-04-20T20:18:07.0830832Z + lsmod 2024-04-20T20:18:07.0847749Z Module Size Used by 2024-04-20T20:18:07.0849153Z ib_core 393216 0 2024-04-20T20:18:07.0849686Z nvidia_modeset 1302528 0 2024-04-20T20:18:07.0850152Z veth 32768 0 2024-04-20T20:18:07.0850591Z nvidia_uvm 1540096 0 2024-04-20T20:18:07.0851169Z nvidia 56455168 2 nvidia_uvm,nvidia_modeset 2024-04-20T20:18:07.0851763Z drm 618496 1 nvidia 2024-04-20T20:18:07.0852250Z xt_conntrack 16384 1 2024-04-20T20:18:07.0852688Z xt_MASQUERADE 20480 1 2024-04-20T20:18:07.0853293Z xfrm_user 40960 1 2024-04-20T20:18:07.0853760Z xfrm_algo 16384 1 xfrm_user 2024-04-20T20:18:07.0854258Z xt_addrtype 16384 2 2024-04-20T20:18:07.0854692Z iptable_filter 16384 1 2024-04-20T20:18:07.0855142Z iptable_nat 16384 1 2024-04-20T20:18:07.0855673Z nf_nat 49152 2 iptable_nat,xt_MASQUERADE 2024-04-20T20:18:07.0856249Z bpfilter 16384 0 2024-04-20T20:18:07.0856686Z br_netfilter 28672 0 2024-04-20T20:18:07.0857180Z bridge 307200 1 br_netfilter 2024-04-20T20:18:07.0857902Z stp 16384 1 bridge 2024-04-20T20:18:07.0858406Z llc 16384 2 bridge,stp 2024-04-20T20:18:07.0858887Z aufs 270336 0 2024-04-20T20:18:07.0859323Z overlay 151552 0 2024-04-20T20:18:07.0859756Z nls_iso8859_1 16384 1 2024-04-20T20:18:07.0860185Z dm_multipath 40960 0 2024-04-20T20:18:07.0860626Z scsi_dh_rdac 16384 0 2024-04-20T20:18:07.0861070Z scsi_dh_emc 16384 0 2024-04-20T20:18:07.0861560Z scsi_dh_alua 20480 0 2024-04-20T20:18:07.0861982Z binfmt_misc 24576 1 2024-04-20T20:18:07.0862432Z crct10dif_pclmul 16384 1 2024-04-20T20:18:07.0862875Z crc32_pclmul 16384 0 2024-04-20T20:18:07.0863307Z virtio_net 61440 0 2024-04-20T20:18:07.0863736Z ghash_clmulni_intel 16384 0 2024-04-20T20:18:07.0864282Z net_failover 20480 1 virtio_net 2024-04-20T20:18:07.0865070Z aesni_intel 376832 0 2024-04-20T20:18:07.0865900Z psmouse 176128 0 2024-04-20T20:18:07.0866702Z input_leds 16384 0 2024-04-20T20:18:07.0867198Z failover 16384 1 net_failover 2024-04-20T20:18:07.0867757Z crypto_simd 16384 1 aesni_intel 2024-04-20T20:18:07.0868635Z cryptd 24576 2 crypto_simd,ghash_clmulni_intel 2024-04-20T20:18:07.0869554Z serio_raw 20480 0 2024-04-20T20:18:07.0870014Z sch_fq_codel 24576 13 2024-04-20T20:18:07.0870464Z efi_pstore 16384 0 2024-04-20T20:18:07.0870893Z virtio_rng 16384 0 2024-04-20T20:18:07.0871500Z ip_tables 32768 2 iptable_filter,iptable_nat 2024-04-20T20:18:07.0872473Z x_tables 53248 6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE 2024-04-20T20:18:07.0873328Z autofs4 49152 2 2024-04-20T20:18:07.0873749Z + modinfo nvidia 2024-04-20T20:18:07.0874541Z filename: /lib/modules/5.15.0-1041-gcp/kernel/drivers/video/nvidia.ko 2024-04-20T20:18:07.0875314Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-04-20T20:18:07.0875901Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-04-20T20:18:07.0876486Z alias: char-major-195-* 2024-04-20T20:18:07.0876944Z version: 535.54.03 2024-04-20T20:18:07.0877353Z supported: external 2024-04-20T20:18:07.0877757Z license: NVIDIA 2024-04-20T20:18:07.0878162Z srcversion: 415DC1001E7D8AD79B6207C 2024-04-20T20:18:07.0878732Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-04-20T20:18:07.0879332Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-04-20T20:18:07.0879930Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-04-20T20:18:07.0880450Z depends: drm 2024-04-20T20:18:07.0880820Z retpoline: Y 2024-04-20T20:18:07.0881217Z name: nvidia 2024-04-20T20:18:07.0881823Z vermagic: 5.15.0-1041-gcp SMP mod_unload modversions 2024-04-20T20:18:07.0882756Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-04-20T20:18:07.0883576Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-04-20T20:18:07.0884313Z parm: NVreg_ResmanDebugLevel:int 2024-04-20T20:18:07.0884861Z parm: NVreg_RmLogonRC:int 2024-04-20T20:18:07.0885371Z parm: NVreg_ModifyDeviceFiles:int 2024-04-20T20:18:07.0885924Z parm: NVreg_DeviceFileUID:int 2024-04-20T20:18:07.0886454Z parm: NVreg_DeviceFileGID:int 2024-04-20T20:18:07.0886969Z parm: NVreg_DeviceFileMode:int 2024-04-20T20:18:07.0887612Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-04-20T20:18:07.0888302Z parm: NVreg_UsePageAttributeTable:int 2024-04-20T20:18:07.0888888Z parm: NVreg_EnablePCIeGen3:int 2024-04-20T20:18:07.0889389Z parm: NVreg_EnableMSI:int 2024-04-20T20:18:07.0889883Z parm: NVreg_TCEBypassMode:int 2024-04-20T20:18:07.0890434Z parm: NVreg_EnableStreamMemOPs:int 2024-04-20T20:18:07.0891090Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-04-20T20:18:07.0891972Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-04-20T20:18:07.0892672Z parm: NVreg_EnableS0ixPowerManagement:int 2024-04-20T20:18:07.0893594Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-04-20T20:18:07.0894349Z parm: NVreg_DynamicPowerManagement:int 2024-04-20T20:18:07.0895100Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-04-20T20:18:07.0895843Z parm: NVreg_EnableGpuFirmware:int 2024-04-20T20:18:07.0896437Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-04-20T20:18:07.0897092Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-04-20T20:18:07.0897756Z parm: NVreg_EnableUserNUMAManagement:int 2024-04-20T20:18:07.0898362Z parm: NVreg_MemoryPoolSize:int 2024-04-20T20:18:07.0898924Z parm: NVreg_KMallocHeapMaxSize:int 2024-04-20T20:18:07.0899515Z parm: NVreg_VMallocHeapMaxSize:int 2024-04-20T20:18:07.0900079Z parm: NVreg_IgnoreMMIOCheck:int 2024-04-20T20:18:07.0900622Z parm: NVreg_NvLinkDisable:int 2024-04-20T20:18:07.0901280Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-04-20T20:18:07.0901926Z parm: NVreg_RegisterPCIDriver:int 2024-04-20T20:18:07.0902492Z parm: NVreg_EnableResizableBar:int 2024-04-20T20:18:07.0903088Z parm: NVreg_EnableDbgBreakpoint:int 2024-04-20T20:18:07.0903674Z parm: NVreg_RegistryDwords:charp 2024-04-20T20:18:07.0904261Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-04-20T20:18:07.0904851Z parm: NVreg_RmMsg:charp 2024-04-20T20:18:07.0905342Z parm: NVreg_GpuBlacklist:charp 2024-04-20T20:18:07.0905904Z parm: NVreg_TemporaryFilePath:charp 2024-04-20T20:18:07.0906460Z parm: NVreg_ExcludedGpus:charp 2024-04-20T20:18:07.0907013Z parm: NVreg_DmaRemapPeerMmio:int 2024-04-20T20:18:07.0907588Z parm: NVreg_RmNvlinkBandwidth:charp 2024-04-20T20:18:07.0908145Z parm: rm_firmware_active:charp 2024-04-20T20:18:07.0908619Z + set +e 2024-04-20T20:18:07.0908999Z + nvidia-smi 2024-04-20T20:18:07.1042157Z Sat Apr 20 20:18:07 2024 2024-04-20T20:18:07.1043177Z +---------------------------------------------------------------------------------------+ 2024-04-20T20:18:07.1044238Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2024-04-20T20:18:07.1045227Z |-----------------------------------------+----------------------+----------------------+ 2024-04-20T20:18:07.1046229Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-04-20T20:18:07.1047340Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-04-20T20:18:07.1048170Z | | | MIG M. | 2024-04-20T20:18:07.1049104Z |=========================================+======================+======================| 2024-04-20T20:18:07.1152851Z | 0 NVIDIA A100-SXM4-40GB On | 00000000:00:04.0 Off | 0 | 2024-04-20T20:18:07.1154316Z | N/A 30C P0 46W / 400W | 4MiB / 40960MiB | 0% Default | 2024-04-20T20:18:07.1155114Z | | | Disabled | 2024-04-20T20:18:07.1155990Z +-----------------------------------------+----------------------+----------------------+ 2024-04-20T20:18:07.1156952Z 2024-04-20T20:18:07.1158011Z +---------------------------------------------------------------------------------------+ 2024-04-20T20:18:07.1158799Z | Processes: | 2024-04-20T20:18:07.1159657Z | GPU GI CI PID Type Process name GPU Memory | 2024-04-20T20:18:07.1160524Z | ID ID Usage | 2024-04-20T20:18:07.1161578Z |=======================================================================================| 2024-04-20T20:18:07.1162393Z | No running processes found | 2024-04-20T20:18:07.1163535Z +---------------------------------------------------------------------------------------+ 2024-04-20T20:18:07.4855965Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-04-20T20:18:07.5004235Z NVIDIA A100-SXM4-40GB 2024-04-20T20:18:07.5035102Z + NVIDIA_SMI_STATUS=0 2024-04-20T20:18:07.5036190Z + '[' 0 -eq 0 ']' 2024-04-20T20:18:07.5036950Z + echo 'INFO: Ignoring allowed status 0' 2024-04-20T20:18:07.5037757Z + set -e 2024-04-20T20:18:07.5038330Z INFO: Ignoring allowed status 0 2024-04-20T20:18:07.5045059Z == Installing nvidia container toolkit for ubuntu20.04 == 2024-04-20T20:18:07.5053277Z ++ dpkg-query -W '--showformat=${db:Status-Status}' nvidia-docker2 2024-04-20T20:18:07.5157038Z + status=installed 2024-04-20T20:18:07.5157622Z + '[' '!' 0 = 0 ']' 2024-04-20T20:18:07.5158089Z + '[' '!' installed = installed ']' 2024-04-20T20:18:08.0175107Z Command completed after 1 attempt(s). 2024-04-20T20:18:08.0257419Z ##[group]Run sudo nvidia-smi -pm 1 2024-04-20T20:18:08.0257976Z sudo nvidia-smi -pm 1 2024-04-20T20:18:08.0258454Z sudo nvidia-smi -ac 1215,1410 2024-04-20T20:18:08.0258949Z nvidia-smi 2024-04-20T20:18:08.0272135Z shell: /usr/bin/bash -e {0} 2024-04-20T20:18:08.0272565Z env: 2024-04-20T20:18:08.0272886Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:08.0273435Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:08.0274006Z ##[endgroup] 2024-04-20T20:18:08.0573593Z Persistence mode is already Enabled for GPU 00000000:00:04.0. 2024-04-20T20:18:08.0574274Z All done. 2024-04-20T20:18:08.0963003Z Applications clocks set to "(MEM 1215, SM 1410)" for GPU 00000000:00:04.0 2024-04-20T20:18:08.0963813Z All done. 2024-04-20T20:18:08.1144717Z Sat Apr 20 20:18:08 2024 2024-04-20T20:18:08.1146256Z +---------------------------------------------------------------------------------------+ 2024-04-20T20:18:08.1148158Z | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2024-04-20T20:18:08.1149305Z |-----------------------------------------+----------------------+----------------------+ 2024-04-20T20:18:08.1150290Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-04-20T20:18:08.1151413Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-04-20T20:18:08.1152273Z | | | MIG M. | 2024-04-20T20:18:08.1152953Z |=========================================+======================+======================| 2024-04-20T20:18:08.1217850Z | 0 NVIDIA A100-SXM4-40GB On | 00000000:00:04.0 Off | 0 | 2024-04-20T20:18:08.1219454Z | N/A 31C P0 53W / 400W | 4MiB / 40960MiB | 0% Default | 2024-04-20T20:18:08.1220279Z | | | Disabled | 2024-04-20T20:18:08.1221777Z +-----------------------------------------+----------------------+----------------------+ 2024-04-20T20:18:08.1222588Z 2024-04-20T20:18:08.1223401Z +---------------------------------------------------------------------------------------+ 2024-04-20T20:18:08.1224191Z | Processes: | 2024-04-20T20:18:08.1225063Z | GPU GI CI PID Type Process name GPU Memory | 2024-04-20T20:18:08.1226110Z | ID ID Usage | 2024-04-20T20:18:08.1226798Z |=======================================================================================| 2024-04-20T20:18:08.1227978Z | No running processes found | 2024-04-20T20:18:08.1228972Z +---------------------------------------------------------------------------------------+ 2024-04-20T20:18:08.3860270Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-20T20:18:08.3861432Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-04-20T20:18:08.3862290Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-04-20T20:18:08.3863089Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-04-20T20:18:08.3876618Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:08.3877232Z env: 2024-04-20T20:18:08.3877562Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:08.3878108Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:08.3878687Z ##[endgroup] 2024-04-20T20:18:09.5258834Z Requirement already satisfied: psutil==5.9.1 in /home/ubuntu/.local/lib/python3.8/site-packages (5.9.1) 2024-04-20T20:18:09.5346549Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ubuntu/.local/lib/python3.8/site-packages (11.525.84) 2024-04-20T20:18:09.7602904Z Prepare all required actions 2024-04-20T20:18:09.7604379Z Getting action download info 2024-04-20T20:18:09.8911359Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-04-20T20:18:10.3740791Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-04-20T20:18:10.7209977Z ##[group]Run ./.github/actions/download-build-artifacts 2024-04-20T20:18:10.7210555Z with: 2024-04-20T20:18:10.7210949Z name: linux-focal-cuda12.1-py3.10-gcc9-sm80 2024-04-20T20:18:10.7211508Z s3-bucket: gha-artifacts 2024-04-20T20:18:10.7211946Z env: 2024-04-20T20:18:10.7212251Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:10.7212788Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:10.7213707Z ##[endgroup] 2024-04-20T20:18:10.7255124Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-20T20:18:10.7255737Z with: 2024-04-20T20:18:10.7256184Z name: linux-focal-cuda12.1-py3.10-gcc9-sm80 2024-04-20T20:18:10.7256740Z s3-bucket: gha-artifacts 2024-04-20T20:18:10.7257158Z region: us-east-1 2024-04-20T20:18:10.7257501Z env: 2024-04-20T20:18:10.7257803Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:10.7258340Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:10.7258922Z ##[endgroup] 2024-04-20T20:18:11.3226330Z (node:953232) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-20T20:18:11.3227256Z 2024-04-20T20:18:11.3227588Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-20T20:18:11.3228482Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-20T20:18:11.3230225Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-20T20:18:11.5062857Z Found 1 objects with prefix pytorch/pytorch/8763335203/linux-focal-cuda12.1-py3.10-gcc9-sm80/ 2024-04-20T20:18:11.5064953Z Starting download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-20T20:18:26.3153689Z Finished download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-04-20T20:18:26.3163567Z Artifact download has finished successfully 2024-04-20T20:18:26.3439832Z ##[group]Run unzip -o artifacts.zip 2024-04-20T20:18:26.3440361Z unzip -o artifacts.zip 2024-04-20T20:18:26.3455270Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:26.3455894Z env: 2024-04-20T20:18:26.3456222Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:26.3456750Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:26.3457338Z ##[endgroup] 2024-04-20T20:18:26.3515981Z Archive: artifacts.zip 2024-04-20T20:18:26.3519386Z creating: dist/ 2024-04-20T20:18:28.7142194Z inflating: dist/torch-2.4.0a0+gitf20e3ae-cp310-cp310-linux_x86_64.whl 2024-04-20T20:18:28.7143464Z creating: build/custom_test_artifacts/ 2024-04-20T20:18:28.7144260Z creating: build/custom_test_artifacts/custom-op-build/ 2024-04-20T20:18:28.7145165Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-04-20T20:18:28.7146299Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-04-20T20:18:28.7151740Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-20T20:18:28.7153059Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-04-20T20:18:28.7154296Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-20T20:18:28.7155580Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-20T20:18:28.7157260Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-20T20:18:28.7158872Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-20T20:18:28.7161368Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-20T20:18:28.7164016Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-20T20:18:28.7166324Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-20T20:18:28.7168592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-20T20:18:28.7170123Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-20T20:18:28.7171612Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-20T20:18:28.7173261Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-20T20:18:28.7174889Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-20T20:18:28.7176477Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-20T20:18:28.7177829Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-20T20:18:28.7179141Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-20T20:18:28.7215395Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-20T20:18:28.7261485Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-20T20:18:28.7263439Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-20T20:18:28.7315831Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-20T20:18:28.7319551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-20T20:18:28.7322613Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-20T20:18:28.7324941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-20T20:18:28.7326847Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-20T20:18:28.7328690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-20T20:18:28.7330529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-20T20:18:28.7332362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-20T20:18:28.7334600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-20T20:18:28.7336330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-20T20:18:28.7337955Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-20T20:18:28.7339556Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-20T20:18:28.7341179Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-20T20:18:28.7342743Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-20T20:18:28.7344326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-20T20:18:28.7395406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-20T20:18:28.7467534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-20T20:18:28.7470727Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-20T20:18:28.7473217Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-04-20T20:18:28.7474358Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-04-20T20:18:28.7475532Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-04-20T20:18:28.7476773Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-04-20T20:18:28.7478101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-04-20T20:18:28.7479628Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-04-20T20:18:28.7481056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-04-20T20:18:28.7482402Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-04-20T20:18:28.7483809Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-04-20T20:18:28.7485215Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-04-20T20:18:28.7486654Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-04-20T20:18:28.7488054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-04-20T20:18:28.7489431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-04-20T20:18:28.7498235Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-04-20T20:18:28.7657521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-04-20T20:18:28.7659944Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-04-20T20:18:28.7661677Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-04-20T20:18:28.7663901Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-04-20T20:18:28.7665432Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-04-20T20:18:28.7666902Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-04-20T20:18:28.7668380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-04-20T20:18:28.7670129Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-04-20T20:18:28.7671616Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-04-20T20:18:28.7673097Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-04-20T20:18:28.7674559Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-04-20T20:18:28.7688459Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-04-20T20:18:28.7786456Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-04-20T20:18:28.7788535Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-20T20:18:28.7790426Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-04-20T20:18:28.7792607Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-04-20T20:18:28.7793793Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-04-20T20:18:28.7795256Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-04-20T20:18:28.7797407Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-04-20T20:18:28.7798747Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-04-20T20:18:28.7799727Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-04-20T20:18:28.7800763Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-04-20T20:18:28.7931627Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-04-20T20:18:28.8006300Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-04-20T20:18:28.8007597Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-04-20T20:18:28.8008485Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-04-20T20:18:28.8009514Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-04-20T20:18:28.8016450Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-20T20:18:28.8017962Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-04-20T20:18:28.8019401Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-20T20:18:28.8021643Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-20T20:18:28.8022990Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-20T20:18:28.8024548Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-20T20:18:28.8026350Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-20T20:18:28.8027914Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-20T20:18:28.8029291Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-20T20:18:28.8031689Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-20T20:18:28.8033202Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-20T20:18:28.8034676Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-20T20:18:28.8036196Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-20T20:18:28.8037690Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-20T20:18:28.8039467Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-20T20:18:28.8040790Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-20T20:18:28.8042079Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-20T20:18:28.8079146Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-20T20:18:28.8125249Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-20T20:18:28.8127234Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-20T20:18:28.8179468Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-20T20:18:28.8182668Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-20T20:18:28.8185793Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-20T20:18:28.8189194Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-20T20:18:28.8191044Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-20T20:18:28.8192868Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-20T20:18:28.8194699Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-20T20:18:28.8196586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-20T20:18:28.8198371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-20T20:18:28.8200031Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-20T20:18:28.8201644Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-20T20:18:28.8203254Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-20T20:18:28.8204858Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-20T20:18:28.8206436Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-20T20:18:28.8208025Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-20T20:18:28.8260816Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-20T20:18:28.8333102Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-20T20:18:28.8336256Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-20T20:18:28.8338728Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-04-20T20:18:28.8339923Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-04-20T20:18:28.8341071Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-04-20T20:18:28.8342286Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-04-20T20:18:28.8343669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-04-20T20:18:28.8345454Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-04-20T20:18:28.8347004Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-04-20T20:18:28.8348410Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-04-20T20:18:28.8349846Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-04-20T20:18:28.8351286Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-04-20T20:18:28.8352722Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-04-20T20:18:28.8354188Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-04-20T20:18:28.8355611Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-04-20T20:18:28.8363977Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-04-20T20:18:28.8440207Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-04-20T20:18:28.8442507Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-20T20:18:28.8444255Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-04-20T20:18:28.8446500Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-04-20T20:18:28.8447664Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-04-20T20:18:28.8448794Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-04-20T20:18:28.8449942Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-04-20T20:18:28.8451026Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-04-20T20:18:28.8452006Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-04-20T20:18:28.8453182Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-04-20T20:18:28.8504570Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-04-20T20:18:28.8505689Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-04-20T20:18:28.8506714Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-04-20T20:18:28.8507851Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-04-20T20:18:28.8514113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-04-20T20:18:28.8515598Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-04-20T20:18:28.8517549Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-04-20T20:18:28.8519673Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-04-20T20:18:28.8521314Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-04-20T20:18:28.8524058Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-04-20T20:18:28.8526753Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-04-20T20:18:28.8529275Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-04-20T20:18:28.8530637Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-04-20T20:18:28.8532209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-04-20T20:18:28.8534116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-04-20T20:18:28.8535961Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-04-20T20:18:28.8537579Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-04-20T20:18:28.8539232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-04-20T20:18:28.8540829Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-04-20T20:18:28.8542259Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-04-20T20:18:28.8543635Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-04-20T20:18:28.8577309Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-04-20T20:18:28.8623689Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-04-20T20:18:28.8626024Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-04-20T20:18:28.8678430Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-04-20T20:18:28.8682123Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-04-20T20:18:28.8685629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-04-20T20:18:28.8688157Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-04-20T20:18:28.8690114Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-04-20T20:18:28.8692048Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-04-20T20:18:28.8694152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-04-20T20:18:28.8696073Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-04-20T20:18:28.8698002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-04-20T20:18:28.8699778Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-04-20T20:18:28.8701475Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-04-20T20:18:28.8703161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-04-20T20:18:28.8704865Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-04-20T20:18:28.8706548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-04-20T20:18:28.8708232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-04-20T20:18:28.8758862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-04-20T20:18:28.8831598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-04-20T20:18:28.8834849Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-04-20T20:18:28.8837354Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-04-20T20:18:28.8838845Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-04-20T20:18:28.8840085Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-04-20T20:18:28.8841823Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-04-20T20:18:28.8843504Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-04-20T20:18:28.8845172Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-04-20T20:18:28.8846842Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-04-20T20:18:28.8848348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-04-20T20:18:28.8849889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-04-20T20:18:28.8851622Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-04-20T20:18:28.8853381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-04-20T20:18:28.8854925Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-04-20T20:18:28.8856499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-04-20T20:18:28.8858138Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-04-20T20:18:28.8991076Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-04-20T20:18:28.8994139Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-04-20T20:18:28.8995841Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-04-20T20:18:28.8997675Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-04-20T20:18:28.8999345Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-04-20T20:18:28.9000924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-04-20T20:18:28.9002553Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-04-20T20:18:28.9004185Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-04-20T20:18:28.9005831Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-04-20T20:18:28.9008492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-04-20T20:18:28.9010186Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-04-20T20:18:28.9019757Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-04-20T20:18:28.9085566Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-04-20T20:18:28.9088165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-04-20T20:18:28.9090150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-04-20T20:18:28.9092195Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-04-20T20:18:28.9093775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-04-20T20:18:28.9095511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-04-20T20:18:28.9096761Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-04-20T20:18:28.9097944Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-04-20T20:18:28.9099014Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-04-20T20:18:28.9100106Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-04-20T20:18:28.9213820Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-04-20T20:18:28.9263644Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-04-20T20:18:28.9264470Z creating: build/lib/ 2024-04-20T20:18:28.9369555Z inflating: build/lib/libprotobuf-lite.a 2024-04-20T20:18:28.9915017Z inflating: build/lib/libprotobuf.a 2024-04-20T20:18:28.9925289Z inflating: build/lib/libpthreadpool.a 2024-04-20T20:18:28.9935702Z inflating: build/lib/libcpuinfo.a 2024-04-20T20:18:28.9944495Z inflating: build/lib/libcpuinfo_internals.a 2024-04-20T20:18:28.9945399Z inflating: build/lib/libclog.a 2024-04-20T20:18:28.9966170Z inflating: build/lib/libnnpack.a 2024-04-20T20:18:28.9968713Z inflating: build/lib/libnnpack_reference_layers.a 2024-04-20T20:18:29.0045779Z inflating: build/lib/libgtest.a 2024-04-20T20:18:29.0135047Z inflating: build/lib/libbenchmark.a 2024-04-20T20:18:29.0212184Z inflating: build/lib/libasmjit.a 2024-04-20T20:18:29.0220930Z inflating: build/lib/libittnotify.a 2024-04-20T20:18:29.0254196Z inflating: build/lib/libtensorpipe_uv.a 2024-04-20T20:18:29.0406045Z inflating: build/lib/libgloo.a 2024-04-20T20:18:29.0431994Z inflating: build/lib/libfmt.a 2024-04-20T20:18:29.0539105Z inflating: build/lib/libc10.so 2024-04-20T20:18:29.0540593Z inflating: build/lib/libcaffe2_nvrtc.so 2024-04-20T20:18:29.0541761Z inflating: build/lib/libfoxi_loader.a 2024-04-20T20:18:29.0542889Z inflating: build/lib/libtorch_global_deps.so 2024-04-20T20:18:29.0558867Z inflating: build/lib/libqnnpack.a 2024-04-20T20:18:29.1163006Z inflating: build/lib/libprotoc.a 2024-04-20T20:18:29.1185797Z inflating: build/lib/libpytorch_qnnpack.a 2024-04-20T20:18:29.1206930Z inflating: build/lib/libgmock.a 2024-04-20T20:18:29.1207960Z inflating: build/lib/libgtest_main.a 2024-04-20T20:18:29.1208833Z inflating: build/lib/libbenchmark_main.a 2024-04-20T20:18:29.1672112Z inflating: build/lib/libgloo_cuda.a 2024-04-20T20:18:30.3006111Z inflating: build/lib/libdnnl.a 2024-04-20T20:18:30.3687920Z inflating: build/lib/libtensorpipe.a 2024-04-20T20:18:30.3752097Z inflating: build/lib/libc10_cuda.so 2024-04-20T20:18:30.3753118Z inflating: build/lib/libgmock_main.a 2024-04-20T20:18:30.5249902Z inflating: build/lib/libfbgemm.a 2024-04-20T20:18:30.5552994Z inflating: build/lib/libtensorpipe_cuda.a 2024-04-20T20:18:30.6173776Z inflating: build/lib/libkineto.a 2024-04-20T20:18:30.6229589Z inflating: build/lib/libcaffe2_protos.a 2024-04-20T20:18:30.6279853Z inflating: build/lib/libonnx_proto.a 2024-04-20T20:18:30.6502707Z inflating: build/lib/libXNNPACK.a 2024-04-20T20:18:30.7342642Z inflating: build/lib/libonnx.a 2024-04-20T20:18:33.5494752Z inflating: build/lib/libtorch_cpu.so 2024-04-20T20:18:33.5498920Z inflating: build/lib/libunbox_lib.a 2024-04-20T20:18:33.5503542Z inflating: build/lib/libshm.so 2024-04-20T20:18:36.0561752Z inflating: build/lib/libtorch_cuda.so 2024-04-20T20:18:36.0562402Z inflating: build/lib/libtorch.so 2024-04-20T20:18:37.0519008Z inflating: build/lib/libtorch_cuda_linalg.so 2024-04-20T20:18:37.0520014Z inflating: build/lib/libc10d_cuda_test.so 2024-04-20T20:18:37.2777961Z inflating: build/lib/libtorch_python.so 2024-04-20T20:18:37.2855521Z inflating: build/lib/libtorchbind_test.so 2024-04-20T20:18:37.2880121Z inflating: build/lib/libjitbackend_test.so 2024-04-20T20:18:37.2910103Z inflating: build/lib/libbackend_with_compiler.so 2024-04-20T20:18:37.2951140Z inflating: build/lib/libnnapi_backend.so 2024-04-20T20:18:37.2952065Z creating: build/bin/ 2024-04-20T20:18:37.3011542Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-04-20T20:18:37.3073503Z inflating: build/bin/c10_DeviceGuard_test 2024-04-20T20:18:37.3133171Z inflating: build/bin/c10_Device_test 2024-04-20T20:18:37.3203881Z inflating: build/bin/c10_DispatchKeySet_test 2024-04-20T20:18:37.3268018Z inflating: build/bin/c10_Scalar_test 2024-04-20T20:18:37.3326627Z inflating: build/bin/c10_StreamGuard_test 2024-04-20T20:18:37.3387447Z inflating: build/bin/c10_SymInt_test 2024-04-20T20:18:37.3452476Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-04-20T20:18:37.3519347Z inflating: build/bin/c10_InlineStreamGuard_test 2024-04-20T20:18:37.3586588Z inflating: build/bin/c10_SizesAndStrides_test 2024-04-20T20:18:37.3671592Z inflating: build/bin/c10_cow_test 2024-04-20T20:18:37.3735431Z inflating: build/bin/c10_Bitset_test 2024-04-20T20:18:37.3793608Z inflating: build/bin/c10_ConstexprCrc_test 2024-04-20T20:18:37.3853217Z inflating: build/bin/c10_DeadlockDetection_test 2024-04-20T20:18:37.3913788Z inflating: build/bin/c10_Half_test 2024-04-20T20:18:37.3981122Z inflating: build/bin/c10_LeftRight_test 2024-04-20T20:18:37.4047099Z inflating: build/bin/c10_Metaprogramming_test 2024-04-20T20:18:37.4106894Z inflating: build/bin/c10_Synchronized_test 2024-04-20T20:18:37.4173534Z inflating: build/bin/c10_ThreadLocal_test 2024-04-20T20:18:37.4234946Z inflating: build/bin/c10_TypeIndex_test 2024-04-20T20:18:37.4296359Z inflating: build/bin/c10_TypeList_test 2024-04-20T20:18:37.4354608Z inflating: build/bin/c10_TypeTraits_test 2024-04-20T20:18:37.4416762Z inflating: build/bin/c10_accumulate_test 2024-04-20T20:18:37.4482897Z inflating: build/bin/c10_bfloat16_test 2024-04-20T20:18:37.4543976Z inflating: build/bin/c10_bit_cast_test 2024-04-20T20:18:37.4611272Z inflating: build/bin/c10_complex_math_test 2024-04-20T20:18:37.4678019Z inflating: build/bin/c10_complex_test 2024-04-20T20:18:37.4740947Z inflating: build/bin/c10_exception_test 2024-04-20T20:18:37.4801136Z inflating: build/bin/c10_flags_test 2024-04-20T20:18:37.4861091Z inflating: build/bin/c10_generic_math_test 2024-04-20T20:18:37.4921613Z inflating: build/bin/c10_irange_test 2024-04-20T20:18:37.5117796Z inflating: build/bin/c10_intrusive_ptr_test 2024-04-20T20:18:37.5184481Z inflating: build/bin/c10_logging_test 2024-04-20T20:18:37.5275363Z inflating: build/bin/c10_optional_test 2024-04-20T20:18:37.5350055Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-04-20T20:18:37.5413652Z inflating: build/bin/c10_registry_test 2024-04-20T20:18:37.5593860Z inflating: build/bin/c10_small_vector_test 2024-04-20T20:18:37.5655969Z inflating: build/bin/c10_ssize_test 2024-04-20T20:18:37.5717868Z inflating: build/bin/c10_string_util_test 2024-04-20T20:18:37.5788087Z inflating: build/bin/c10_string_view_test 2024-04-20T20:18:37.5848085Z inflating: build/bin/c10_tempfile_test 2024-04-20T20:18:37.5915001Z inflating: build/bin/c10_typeid_test 2024-04-20T20:18:37.5973227Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-04-20T20:18:37.6512285Z inflating: build/bin/protoc-3.13.0.0 2024-04-20T20:18:37.7047794Z inflating: build/bin/protoc 2024-04-20T20:18:37.7111284Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-04-20T20:18:37.7174679Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-04-20T20:18:37.7238244Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-04-20T20:18:37.7300569Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-04-20T20:18:37.7363561Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-04-20T20:18:37.7426966Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-04-20T20:18:37.7485204Z inflating: build/bin/c10_cuda_CUDATest 2024-04-20T20:18:37.7548647Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-04-20T20:18:37.7947124Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-04-20T20:18:37.8359632Z inflating: build/bin/vec_test_all_types_AVX512 2024-04-20T20:18:37.8789675Z inflating: build/bin/vec_test_all_types_AVX2 2024-04-20T20:18:37.8853352Z inflating: build/bin/FileStoreTest 2024-04-20T20:18:37.8919982Z inflating: build/bin/TCPStoreTest 2024-04-20T20:18:37.8983879Z inflating: build/bin/HashStoreTest 2024-04-20T20:18:37.8999957Z inflating: build/bin/ProcessGroupMPITest 2024-04-20T20:18:37.9064478Z inflating: build/bin/test_edge_op_registration 2024-04-20T20:18:37.9069293Z inflating: build/bin/torch_shm_manager 2024-04-20T20:18:37.9072587Z inflating: build/bin/example_allreduce 2024-04-20T20:18:37.9138834Z inflating: build/bin/test_dist_autograd 2024-04-20T20:18:37.9224141Z inflating: build/bin/test_cpp_rpc 2024-04-20T20:18:37.9227241Z inflating: build/bin/parallel_benchmark 2024-04-20T20:18:37.9307761Z inflating: build/bin/test_mobile_nnc 2024-04-20T20:18:37.9318175Z inflating: build/bin/aot_model_compiler_test 2024-04-20T20:18:37.9732513Z inflating: build/bin/test_lazy 2024-04-20T20:18:38.1149333Z inflating: build/bin/test_api 2024-04-20T20:18:38.1236731Z inflating: build/bin/Dict_test 2024-04-20T20:18:38.1299070Z inflating: build/bin/Dimname_test 2024-04-20T20:18:38.1376081Z inflating: build/bin/MaybeOwned_test 2024-04-20T20:18:38.1444655Z inflating: build/bin/NamedTensor_test 2024-04-20T20:18:38.1515466Z inflating: build/bin/apply_utils_test 2024-04-20T20:18:38.1585851Z inflating: build/bin/atest 2024-04-20T20:18:38.1661256Z inflating: build/bin/basic 2024-04-20T20:18:38.1727553Z inflating: build/bin/broadcast_test 2024-04-20T20:18:38.1787981Z inflating: build/bin/cpu_allocator_test 2024-04-20T20:18:38.1857301Z inflating: build/bin/cpu_generator_test 2024-04-20T20:18:38.1920911Z inflating: build/bin/cpu_profiling_allocator_test 2024-04-20T20:18:38.2033075Z inflating: build/bin/cpu_rng_test 2024-04-20T20:18:38.2092560Z inflating: build/bin/dispatch_key_set_test 2024-04-20T20:18:38.2152775Z inflating: build/bin/dlconvertor_test 2024-04-20T20:18:38.2222219Z inflating: build/bin/extension_backend_test 2024-04-20T20:18:38.2288294Z inflating: build/bin/half_test 2024-04-20T20:18:38.2403001Z inflating: build/bin/ivalue_test 2024-04-20T20:18:38.2463183Z inflating: build/bin/lazy_tensor_test 2024-04-20T20:18:38.2528336Z inflating: build/bin/math_kernel_test 2024-04-20T20:18:38.2593401Z inflating: build/bin/memory_format_test 2024-04-20T20:18:38.2657541Z inflating: build/bin/memory_overlapping_test 2024-04-20T20:18:38.2721241Z inflating: build/bin/mobile_memory_cleanup 2024-04-20T20:18:38.2788800Z inflating: build/bin/native_test 2024-04-20T20:18:38.2849056Z inflating: build/bin/operator_name_test 2024-04-20T20:18:38.2911091Z inflating: build/bin/operators_test 2024-04-20T20:18:38.2973459Z inflating: build/bin/packedtensoraccessor_test 2024-04-20T20:18:38.3054152Z inflating: build/bin/pow_test 2024-04-20T20:18:38.3123091Z inflating: build/bin/quantized_test 2024-04-20T20:18:38.3183125Z inflating: build/bin/reduce_ops_test 2024-04-20T20:18:38.3244489Z inflating: build/bin/reportMemoryUsage_test 2024-04-20T20:18:38.3312392Z inflating: build/bin/scalar_tensor_test 2024-04-20T20:18:38.3382463Z inflating: build/bin/scalar_test 2024-04-20T20:18:38.3444256Z inflating: build/bin/StorageUtils_test 2024-04-20T20:18:38.3507165Z inflating: build/bin/stride_properties_test 2024-04-20T20:18:38.3601544Z inflating: build/bin/tensor_iterator_test 2024-04-20T20:18:38.3666465Z inflating: build/bin/test_parallel 2024-04-20T20:18:38.3669515Z inflating: build/bin/thread_init_test 2024-04-20T20:18:38.3736593Z inflating: build/bin/type_ptr_test 2024-04-20T20:18:38.3808279Z inflating: build/bin/type_test 2024-04-20T20:18:38.3870945Z inflating: build/bin/undefined_tensor_test 2024-04-20T20:18:38.3872165Z inflating: build/bin/verify_api_visibility 2024-04-20T20:18:38.3955013Z inflating: build/bin/legacy_vmap_test 2024-04-20T20:18:38.4016689Z inflating: build/bin/weakref_test 2024-04-20T20:18:38.4078280Z inflating: build/bin/wrapdim_test 2024-04-20T20:18:38.4140145Z inflating: build/bin/xla_tensor_test 2024-04-20T20:18:38.4211751Z inflating: build/bin/IListRef_test 2024-04-20T20:18:38.4338317Z inflating: build/bin/List_test 2024-04-20T20:18:38.4418060Z inflating: build/bin/KernelFunction_test 2024-04-20T20:18:38.4564165Z inflating: build/bin/kernel_function_legacy_test 2024-04-20T20:18:38.4680110Z inflating: build/bin/kernel_function_test 2024-04-20T20:18:38.4834896Z inflating: build/bin/kernel_lambda_legacy_test 2024-04-20T20:18:38.4957816Z inflating: build/bin/kernel_lambda_test 2024-04-20T20:18:38.5030871Z inflating: build/bin/kernel_stackbased_test 2024-04-20T20:18:38.5146269Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-04-20T20:18:38.5207202Z inflating: build/bin/CppSignature_test 2024-04-20T20:18:38.5265982Z inflating: build/bin/op_allowlist_test 2024-04-20T20:18:38.5332809Z inflating: build/bin/backend_fallback_test 2024-04-20T20:18:38.5409206Z inflating: build/bin/inline_container_test 2024-04-20T20:18:38.5771553Z inflating: build/bin/op_registration_test 2024-04-20T20:18:38.5837926Z inflating: build/bin/cuda_apply_test 2024-04-20T20:18:38.5898757Z inflating: build/bin/cuda_allocator_test 2024-04-20T20:18:38.5969573Z inflating: build/bin/cuda_atomic_ops_test 2024-04-20T20:18:38.6034860Z inflating: build/bin/cuda_caching_host_allocator_test 2024-04-20T20:18:38.6118539Z inflating: build/bin/cuda_complex_math_test 2024-04-20T20:18:38.6189266Z inflating: build/bin/cuda_complex_test 2024-04-20T20:18:38.6248272Z inflating: build/bin/cuda_device_test 2024-04-20T20:18:38.6316816Z inflating: build/bin/cuda_cub_test 2024-04-20T20:18:38.6377561Z inflating: build/bin/cuda_dlconvertor_test 2024-04-20T20:18:38.6455919Z inflating: build/bin/cuda_distributions_test 2024-04-20T20:18:38.6523769Z inflating: build/bin/cuda_generator_test 2024-04-20T20:18:38.6583322Z inflating: build/bin/cuda_half_test 2024-04-20T20:18:38.6644812Z inflating: build/bin/cuda_integer_divider_test 2024-04-20T20:18:38.6703945Z inflating: build/bin/cuda_optional_test 2024-04-20T20:18:38.6765586Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-04-20T20:18:38.6828900Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-04-20T20:18:38.6888900Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-04-20T20:18:38.6961409Z inflating: build/bin/cuda_stream_test 2024-04-20T20:18:38.7020389Z inflating: build/bin/cuda_cudnn_test 2024-04-20T20:18:38.7082866Z inflating: build/bin/cuda_vectorized_test 2024-04-20T20:18:38.7100859Z inflating: build/bin/tutorial_tensorexpr 2024-04-20T20:18:38.7179741Z inflating: build/bin/ProcessGroupGlooTest 2024-04-20T20:18:38.7248773Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-04-20T20:18:38.7323974Z inflating: build/bin/ProcessGroupNCCLTest 2024-04-20T20:18:38.7398536Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-04-20T20:18:38.8398881Z inflating: build/bin/test_tensorexpr 2024-04-20T20:18:38.9085653Z inflating: build/bin/test_jit 2024-04-20T20:18:38.9086312Z creating: .additional_ci_files/ 2024-04-20T20:18:38.9129861Z inflating: .additional_ci_files/test-times.json 2024-04-20T20:18:38.9307786Z inflating: .additional_ci_files/test-class-times.json 2024-04-20T20:18:38.9377066Z ##[group]Run rm artifacts.zip 2024-04-20T20:18:38.9377540Z rm artifacts.zip 2024-04-20T20:18:38.9391133Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:38.9391802Z env: 2024-04-20T20:18:38.9392133Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:38.9392666Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:38.9393259Z ##[endgroup] 2024-04-20T20:18:39.0738222Z ##[group]Run df -H 2024-04-20T20:18:39.0738609Z df -H 2024-04-20T20:18:39.0751564Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:39.0752171Z env: 2024-04-20T20:18:39.0752500Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:39.0753034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:39.0753623Z ##[endgroup] 2024-04-20T20:18:39.0803282Z Filesystem Size Used Avail Use% Mounted on 2024-04-20T20:18:39.0804146Z /dev/root 1.1T 236G 831G 23% / 2024-04-20T20:18:39.0804668Z devtmpfs 45G 0 45G 0% /dev 2024-04-20T20:18:39.0805248Z tmpfs 45G 0 45G 0% /dev/shm 2024-04-20T20:18:39.0806299Z tmpfs 9.0G 1.1M 9.0G 1% /run 2024-04-20T20:18:39.0807220Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-04-20T20:18:39.0807808Z tmpfs 45G 0 45G 0% /sys/fs/cgroup 2024-04-20T20:18:39.0808934Z /dev/loop0 16M 16M 0 100% /snap/aws-cli/130 2024-04-20T20:18:39.0809612Z /dev/loop3 59M 59M 0 100% /snap/core18/2812 2024-04-20T20:18:39.0810267Z /dev/loop4 59M 59M 0 100% /snap/core18/2796 2024-04-20T20:18:39.0810899Z /dev/loop9 97M 97M 0 100% /snap/lxd/23991 2024-04-20T20:18:39.0811540Z /dev/loop11 97M 97M 0 100% /snap/lxd/24061 2024-04-20T20:18:39.0812155Z /dev/sda15 110M 5.5M 104M 5% /boot/efi 2024-04-20T20:18:39.0812771Z /dev/loop7 111M 111M 0 100% /snap/core/16574 2024-04-20T20:18:39.0813695Z /dev/loop1 68M 68M 0 100% /snap/core20/2182 2024-04-20T20:18:39.0814390Z /dev/loop13 42M 42M 0 100% /snap/snapd/21184 2024-04-20T20:18:39.0815066Z /dev/loop10 110M 110M 0 100% /snap/core/16928 2024-04-20T20:18:39.0815979Z /dev/loop14 368M 368M 0 100% /snap/google-cloud-sdk/440 2024-04-20T20:18:39.0816707Z /dev/loop5 68M 68M 0 100% /snap/core20/2264 2024-04-20T20:18:39.0817375Z /dev/loop15 41M 41M 0 100% /snap/snapd/21465 2024-04-20T20:18:39.0818195Z /dev/loop8 369M 369M 0 100% /snap/google-cloud-sdk/442 2024-04-20T20:18:39.0876237Z Prepare all required actions 2024-04-20T20:18:39.0876947Z Getting action download info 2024-04-20T20:18:39.2427155Z ##[group]Run ./.github/actions/download-td-artifacts 2024-04-20T20:18:39.2428163Z with: 2024-04-20T20:18:39.2428677Z env: 2024-04-20T20:18:39.2429233Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:39.2430213Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:39.2431243Z ##[endgroup] 2024-04-20T20:18:39.2485582Z ##[group]Run seemethere/download-artifact-s3@v4 2024-04-20T20:18:39.2486212Z with: 2024-04-20T20:18:39.2486509Z name: td_results 2024-04-20T20:18:39.2486886Z s3-bucket: gha-artifacts 2024-04-20T20:18:39.2487298Z region: us-east-1 2024-04-20T20:18:39.2487643Z env: 2024-04-20T20:18:39.2487943Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:39.2488481Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:39.2489063Z ##[endgroup] 2024-04-20T20:18:39.8302583Z (node:953276) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-04-20T20:18:39.8303505Z 2024-04-20T20:18:39.8303833Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-04-20T20:18:39.8304717Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-04-20T20:18:39.8305897Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-04-20T20:18:39.9921295Z Found 0 objects with prefix pytorch/pytorch/8763335203/td_results/ 2024-04-20T20:18:39.9929995Z Artifact download has finished successfully 2024-04-20T20:18:40.0189997Z ##[group]Run mkdir -p .additional_ci_files 2024-04-20T20:18:40.0190585Z mkdir -p .additional_ci_files 2024-04-20T20:18:40.0191269Z mv td_results.json .additional_ci_files/td_results.json 2024-04-20T20:18:40.0204581Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:40.0205187Z env: 2024-04-20T20:18:40.0205506Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:40.0206059Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:40.0206685Z ##[endgroup] 2024-04-20T20:18:40.0271128Z mv: cannot stat 'td_results.json': No such file or directory 2024-04-20T20:18:40.0288566Z ##[error]Process completed with exit code 1. 2024-04-20T20:18:40.0330946Z ##[group]Run .github/scripts/parse_ref.py 2024-04-20T20:18:40.0331517Z .github/scripts/parse_ref.py 2024-04-20T20:18:40.0343678Z shell: /usr/bin/bash -e {0} 2024-04-20T20:18:40.0344100Z env: 2024-04-20T20:18:40.0344404Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:40.0344948Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:40.0345550Z ##[endgroup] 2024-04-20T20:18:40.0671322Z Prepare all required actions 2024-04-20T20:18:40.0720652Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-04-20T20:18:40.0721204Z with: 2024-04-20T20:18:40.0721899Z github-token: *** 2024-04-20T20:18:40.0722248Z env: 2024-04-20T20:18:40.0722546Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:40.0723083Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:40.0723664Z ##[endgroup] 2024-04-20T20:18:40.0742416Z ##[group]Run set -eux 2024-04-20T20:18:40.0742810Z set -eux 2024-04-20T20:18:40.0743526Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-04-20T20:18:40.0756199Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:40.0756799Z env: 2024-04-20T20:18:40.0757118Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:40.0757640Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:40.0758468Z GITHUB_TOKEN: *** 2024-04-20T20:18:40.0758828Z ##[endgroup] 2024-04-20T20:18:40.0794296Z + python3 .github/scripts/get_workflow_job_id.py 8763335203 gh-ci-gcp-a100-11 2024-04-20T20:18:40.7278790Z setting job-id=24052568002 2024-04-20T20:18:40.7279896Z setting job-name=cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:18:40.7546996Z Prepare all required actions 2024-04-20T20:18:40.7547536Z Getting action download info 2024-04-20T20:18:40.9047043Z ##[group]Run ./.github/actions/filter-test-configs 2024-04-20T20:18:40.9047609Z with: 2024-04-20T20:18:40.9048186Z github-token: *** 2024-04-20T20:18:40.9055164Z test-matrix: {"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]} 2024-04-20T20:18:40.9062822Z job-name: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:18:40.9063948Z env: 2024-04-20T20:18:40.9064269Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:40.9064794Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:40.9065380Z ##[endgroup] 2024-04-20T20:18:40.9144877Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-04-20T20:18:40.9145555Z with: 2024-04-20T20:18:40.9145880Z shell: bash 2024-04-20T20:18:40.9146231Z timeout_minutes: 10 2024-04-20T20:18:40.9146615Z max_attempts: 5 2024-04-20T20:18:40.9146965Z retry_wait_seconds: 30 2024-04-20T20:18:40.9147926Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-20T20:18:40.9148973Z polling_interval_seconds: 1 2024-04-20T20:18:40.9149417Z warning_on_retry: true 2024-04-20T20:18:40.9149811Z continue_on_error: false 2024-04-20T20:18:40.9150202Z env: 2024-04-20T20:18:40.9150563Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:40.9151095Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:40.9151915Z GITHUB_TOKEN: *** 2024-04-20T20:18:40.9152281Z ##[endgroup] 2024-04-20T20:18:40.9800578Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-04-20T20:18:42.1099287Z Requirement already satisfied: requests==2.26.0 in /home/ubuntu/.local/lib/python3.8/site-packages (2.26.0) 2024-04-20T20:18:42.1221087Z Requirement already satisfied: pyyaml==6.0.1 in /home/ubuntu/.local/lib/python3.8/site-packages (6.0.1) 2024-04-20T20:18:42.1241477Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /usr/lib/python3/dist-packages (from requests==2.26.0) (2.8) 2024-04-20T20:18:42.1258697Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.26.0) (1.25.8) 2024-04-20T20:18:42.1324149Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ubuntu/.local/lib/python3.8/site-packages (from requests==2.26.0) (2.0.12) 2024-04-20T20:18:42.1347399Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.26.0) (2019.11.28) 2024-04-20T20:18:42.9823389Z Command completed after 1 attempt(s). 2024-04-20T20:18:42.9885576Z ##[group]Run set -x 2024-04-20T20:18:42.9885978Z set -x 2024-04-20T20:18:42.9886384Z  2024-04-20T20:18:42.9887298Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-20T20:18:42.9888140Z # in runner workspace 2024-04-20T20:18:42.9888781Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-04-20T20:18:42.9902417Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:42.9903031Z env: 2024-04-20T20:18:42.9903358Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:42.9903904Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:42.9904473Z ##[endgroup] 2024-04-20T20:18:42.9940941Z + python3 /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-04-20T20:18:43.0213081Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-20T20:18:43.0213738Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-04-20T20:18:43.0214290Z echo "Job name: ${JOB_NAME}" 2024-04-20T20:18:43.0214761Z  2024-04-20T20:18:43.0215418Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-04-20T20:18:43.0216311Z # in runner workspace 2024-04-20T20:18:43.0217008Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-04-20T20:18:43.0217796Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-04-20T20:18:43.0218324Z  --job-name "${JOB_NAME}" \ 2024-04-20T20:18:43.0225174Z  --test-matrix "{"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}" \ 2024-04-20T20:18:43.0232708Z  --pr-number "${PR_NUMBER}" \ 2024-04-20T20:18:43.0233210Z  --tag "${TAG}" \ 2024-04-20T20:18:43.0233662Z  --event-name "${EVENT_NAME}" \ 2024-04-20T20:18:43.0234195Z  --schedule "${SCHEDULE}" \ 2024-04-20T20:18:43.0234702Z  --branch "${HEAD_BRANCH}" 2024-04-20T20:18:43.0247209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:43.0247814Z env: 2024-04-20T20:18:43.0248132Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:43.0248658Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:43.0249507Z GITHUB_TOKEN: *** 2024-04-20T20:18:43.0250293Z JOB_NAME: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:18:43.0251180Z PR_NUMBER: 2024-04-20T20:18:43.0251491Z TAG: 2024-04-20T20:18:43.0251803Z EVENT_NAME: schedule 2024-04-20T20:18:43.0252188Z SCHEDULE: 0 7 * * 1-6 2024-04-20T20:18:43.0252567Z HEAD_BRANCH: 2024-04-20T20:18:43.0253083Z ##[endgroup] 2024-04-20T20:18:43.0286612Z Workflow: inductor-A100-perf-nightly 2024-04-20T20:18:43.0287676Z Job name: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:18:43.3963562Z ##[group]Run echo "Filtered matrix:" 2024-04-20T20:18:43.3964118Z echo "Filtered matrix:" 2024-04-20T20:18:43.3971120Z echo "{"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}" 2024-04-20T20:18:43.3978467Z  2024-04-20T20:18:43.3978783Z echo 2024-04-20T20:18:43.3979212Z echo "Is the current job unstable? False" 2024-04-20T20:18:43.3979765Z  2024-04-20T20:18:43.3980070Z echo 2024-04-20T20:18:43.3980489Z echo "Is keep-going label set? False" 2024-04-20T20:18:43.3981003Z  2024-04-20T20:18:43.3981312Z echo 2024-04-20T20:18:43.3981843Z echo "Renabled issues? " 2024-04-20T20:18:43.3994829Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:43.3995437Z env: 2024-04-20T20:18:43.3995748Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:43.3996342Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:43.3996925Z ##[endgroup] 2024-04-20T20:18:43.4030765Z Filtered matrix: 2024-04-20T20:18:43.4039326Z {include: [{config: inductor_huggingface_perf, shard: 1, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_huggingface_perf, shard: 2, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_huggingface_perf, shard: 3, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 1, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 2, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 3, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 4, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 5, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 1, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 2, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 3, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 4, num_shards: 4, runner: linux.gcp.a100.large}]} 2024-04-20T20:18:43.4046201Z 2024-04-20T20:18:43.4046418Z Is the current job unstable? False 2024-04-20T20:18:43.4046757Z 2024-04-20T20:18:43.4047195Z Is keep-going label set? False 2024-04-20T20:18:43.4047496Z 2024-04-20T20:18:43.4047649Z Renabled issues? 2024-04-20T20:18:43.4099687Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-20T20:18:43.4100552Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-04-20T20:18:43.4112832Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-04-20T20:18:43.4113449Z env: 2024-04-20T20:18:43.4113769Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:43.4114296Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:43.4114887Z JOB_TIMEOUT: 720 2024-04-20T20:18:43.4115238Z ##[endgroup] 2024-04-20T20:18:43.4213978Z ##[group]Run set -x 2024-04-20T20:18:43.4214450Z set -x 2024-04-20T20:18:43.4214802Z  2024-04-20T20:18:43.4215224Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-04-20T20:18:43.4215945Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-04-20T20:18:43.4216634Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-04-20T20:18:43.4217256Z  TEST_COMMAND=.ci/onnx/test.sh 2024-04-20T20:18:43.4217751Z else 2024-04-20T20:18:43.4218134Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-04-20T20:18:43.4218637Z fi 2024-04-20T20:18:43.4218954Z  2024-04-20T20:18:43.4219517Z # detached container should get cleaned up by teardown_ec2_linux 2024-04-20T20:18:43.4220451Z # TODO: Stop building test binaries as part of the build phase 2024-04-20T20:18:43.4221278Z # Used for GPU_FLAG since that doesn't play nice 2024-04-20T20:18:43.4222002Z # shellcheck disable=SC2086,SC2090 2024-04-20T20:18:43.4222553Z container_name=$(docker run \ 2024-04-20T20:18:43.4223058Z  ${GPU_FLAG:-} \ 2024-04-20T20:18:43.4223500Z  -e BUILD_ENVIRONMENT \ 2024-04-20T20:18:43.4223983Z  -e PR_NUMBER \ 2024-04-20T20:18:43.4224396Z  -e GITHUB_ACTIONS \ 2024-04-20T20:18:43.4224863Z  -e GITHUB_REPOSITORY \ 2024-04-20T20:18:43.4225554Z  -e GITHUB_WORKFLOW \ 2024-04-20T20:18:43.4226020Z  -e GITHUB_JOB \ 2024-04-20T20:18:43.4226434Z  -e GITHUB_RUN_ID \ 2024-04-20T20:18:43.4226898Z  -e GITHUB_RUN_NUMBER \ 2024-04-20T20:18:43.4227385Z  -e GITHUB_RUN_ATTEMPT \ 2024-04-20T20:18:43.4227859Z  -e JOB_ID \ 2024-04-20T20:18:43.4228443Z  -e JOB_NAME \ 2024-04-20T20:18:43.4228857Z  -e BASE_SHA \ 2024-04-20T20:18:43.4229263Z  -e BRANCH \ 2024-04-20T20:18:43.4229638Z  -e SHA1 \ 2024-04-20T20:18:43.4230044Z  -e AWS_DEFAULT_REGION \ 2024-04-20T20:18:43.4230524Z  -e IN_WHEEL_TEST \ 2024-04-20T20:18:43.4230971Z  -e SHARD_NUMBER \ 2024-04-20T20:18:43.4231394Z  -e TEST_CONFIG \ 2024-04-20T20:18:43.4231835Z  -e NUM_TEST_SHARDS \ 2024-04-20T20:18:43.4232304Z  -e REENABLED_ISSUES \ 2024-04-20T20:18:43.4232799Z  -e CONTINUE_THROUGH_ERROR \ 2024-04-20T20:18:43.4233294Z  -e VERBOSE_TEST_LOGS \ 2024-04-20T20:18:43.4233771Z  -e NO_TEST_TIMEOUT \ 2024-04-20T20:18:43.4234222Z  -e NO_TD \ 2024-04-20T20:18:43.4234625Z  -e TD_DISTRIBUTED \ 2024-04-20T20:18:43.4235064Z  -e PR_LABELS \ 2024-04-20T20:18:43.4235584Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-04-20T20:18:43.4236143Z  -e SCCACHE_BUCKET \ 2024-04-20T20:18:43.4236604Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-04-20T20:18:43.4237099Z  -e XLA_CUDA \ 2024-04-20T20:18:43.4237580Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-04-20T20:18:43.4238195Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-04-20T20:18:43.4238811Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-04-20T20:18:43.4239440Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-04-20T20:18:43.4240006Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-04-20T20:18:43.4240516Z  -e DASHBOARD_TAG \ 2024-04-20T20:18:43.4241074Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-04-20T20:18:43.4241753Z  --security-opt seccomp=unconfined \ 2024-04-20T20:18:43.4242318Z  --cap-add=SYS_PTRACE \ 2024-04-20T20:18:43.4242786Z  --ipc=host \ 2024-04-20T20:18:43.4243192Z  --shm-size="${SHM_SIZE}" \ 2024-04-20T20:18:43.4243673Z  --tty \ 2024-04-20T20:18:43.4244038Z  --detach \ 2024-04-20T20:18:43.4244458Z  --name="${container_name}" \ 2024-04-20T20:18:43.4244947Z  --user jenkins \ 2024-04-20T20:18:43.4245718Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-04-20T20:18:43.4246420Z  -w /var/lib/jenkins/workspace \ 2024-04-20T20:18:43.4246950Z  "${DOCKER_IMAGE}" 2024-04-20T20:18:43.4247348Z ) 2024-04-20T20:18:43.4247831Z # Propagate download.pytorch.org IP to container 2024-04-20T20:18:43.4248991Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-04-20T20:18:43.4250214Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-04-20T20:18:43.4251348Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-04-20T20:18:43.4264070Z shell: /usr/bin/bash -e {0} 2024-04-20T20:18:43.4264489Z env: 2024-04-20T20:18:43.4264812Z GIT_DEFAULT_BRANCH: main 2024-04-20T20:18:43.4265362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:18:43.4266201Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm80 2024-04-20T20:18:43.4266803Z PR_NUMBER: 2024-04-20T20:18:43.4267182Z GITHUB_REPOSITORY: pytorch/pytorch 2024-04-20T20:18:43.4267740Z GITHUB_WORKFLOW: inductor-A100-perf-nightly 2024-04-20T20:18:43.4268273Z GITHUB_JOB: test 2024-04-20T20:18:43.4268626Z GITHUB_RUN_ID: 8763335203 2024-04-20T20:18:43.4269052Z GITHUB_RUN_NUMBER: 3976 2024-04-20T20:18:43.4269461Z GITHUB_RUN_ATTEMPT: 1 2024-04-20T20:18:43.4269852Z JOB_ID: 24052568002 2024-04-20T20:18:43.4270626Z JOB_NAME: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:18:43.4271589Z BRANCH: main 2024-04-20T20:18:43.4272008Z SHA1: f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:18:43.4272633Z BASE_SHA: f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:18:43.4273394Z TEST_CONFIG: inductor_huggingface_perf 2024-04-20T20:18:43.4273895Z SHARD_NUMBER: 1 2024-04-20T20:18:43.4274259Z NUM_TEST_SHARDS: 3 2024-04-20T20:18:43.4274620Z REENABLED_ISSUES: 2024-04-20T20:18:43.4275022Z CONTINUE_THROUGH_ERROR: False 2024-04-20T20:18:43.4275519Z VERBOSE_TEST_LOGS: False 2024-04-20T20:18:43.4275963Z NO_TEST_TIMEOUT: False 2024-04-20T20:18:43.4276345Z NO_TD: False 2024-04-20T20:18:43.4276698Z TD_DISTRIBUTED: False 2024-04-20T20:18:43.4277201Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-04-20T20:18:43.4277874Z SCCACHE_S3_KEY_PREFIX: inductor-A100-perf-nightly 2024-04-20T20:18:43.4278421Z SHM_SIZE: 2g 2024-04-20T20:18:43.4279817Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:18:43.4281325Z XLA_CUDA: 2024-04-20T20:18:43.4281913Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-04-20T20:18:43.4282664Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-04-20T20:18:43.4283192Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-04-20T20:18:43.4284837Z DASHBOARD_TAG: training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true 2024-04-20T20:18:43.4286791Z HUGGING_FACE_HUB_TOKEN: *** 2024-04-20T20:18:43.4287223Z ##[endgroup] 2024-04-20T20:18:43.4320074Z + [[ inductor_huggingface_perf == \m\u\l\t\i\g\p\u ]] 2024-04-20T20:18:43.4321303Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *onnx* ]] 2024-04-20T20:18:43.4321943Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-04-20T20:18:43.4331482Z +++ nproc --ignore=2 2024-04-20T20:18:43.4350682Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=10 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_8763335203 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/weiwangmeta/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:47c5092959bf58e951a46327442e23a676d2687b 2024-04-20T20:18:47.3001146Z + container_name=3755bfd73a941b8a18949c2153815b96befb951b5342ff8426f702914f4e43dc 2024-04-20T20:18:47.3002724Z + grep download.pytorch.org /etc/hosts 2024-04-20T20:18:47.3005357Z + docker exec -i 3755bfd73a941b8a18949c2153815b96befb951b5342ff8426f702914f4e43dc sudo bash -c '/bin/cat >> /etc/hosts' 2024-04-20T20:18:47.3843063Z + echo DOCKER_CONTAINER_ID=3755bfd73a941b8a18949c2153815b96befb951b5342ff8426f702914f4e43dc 2024-04-20T20:18:47.3847956Z ++ echo dist/torch-2.4.0a0+gitf20e3ae-cp310-cp310-linux_x86_64.whl 2024-04-20T20:18:47.3852083Z + docker exec -t 3755bfd73a941b8a18949c2153815b96befb951b5342ff8426f702914f4e43dc sh -c 'pip install dist/torch-2.4.0a0+gitf20e3ae-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-04-20T20:18:47.8923479Z Processing ./dist/torch-2.4.0a0+gitf20e3ae-cp310-cp310-linux_x86_64.whl 2024-04-20T20:18:48.9555234Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitf20e3ae) (3.13.1) 2024-04-20T20:18:48.9560773Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitf20e3ae) (4.11.0) 2024-04-20T20:18:48.9564561Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitf20e3ae) (1.12) 2024-04-20T20:18:48.9567341Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitf20e3ae) (2.8.8) 2024-04-20T20:18:48.9571579Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitf20e3ae) (3.1.3) 2024-04-20T20:18:48.9575491Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitf20e3ae) (2024.2.0) 2024-04-20T20:18:48.9587730Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitf20e3ae) (3.3.0) 2024-04-20T20:18:48.9659824Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitf20e3ae) (1.21.2) 2024-04-20T20:18:49.0170372Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitf20e3ae) (2.1.5) 2024-04-20T20:18:49.0359672Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitf20e3ae) (1.3.0) 2024-04-20T20:18:50.5783987Z Installing collected packages: torch 2024-04-20T20:19:02.2290637Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-04-20T20:19:02.2292461Z timm 0.9.7 requires torchvision, which is not installed. 2024-04-20T20:19:02.2293435Z Successfully installed torch-2.4.0a0+gitf20e3ae 2024-04-20T20:19:02.3217565Z ++ dirname .ci/pytorch/test.sh 2024-04-20T20:19:02.3225164Z + source .ci/pytorch/common.sh 2024-04-20T20:19:02.3230106Z +++ dirname .ci/pytorch/common.sh 2024-04-20T20:19:02.3239016Z ++ source .ci/pytorch/common_utils.sh 2024-04-20T20:19:02.3241944Z +++ declare -f -t trap_add 2024-04-20T20:19:02.3249979Z ++ set -ex 2024-04-20T20:19:02.3251067Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *rocm* ]] 2024-04-20T20:19:02.3251714Z ++ BUILD_TEST_LIBTORCH=0 2024-04-20T20:19:02.3255907Z ++ stat -c %u /var/lib/jenkins/workspace 2024-04-20T20:19:02.3272067Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-04-20T20:19:02.3272979Z + trap_add cleanup_workspace EXIT 2024-04-20T20:19:02.3273496Z + trap_add_cmd=cleanup_workspace 2024-04-20T20:19:02.3273944Z + shift 2024-04-20T20:19:02.3274257Z + for trap_add_name in "$@" 2024-04-20T20:19:02.3283777Z +++ trap -p EXIT 2024-04-20T20:19:02.3286680Z ++ eval 'extract_trap_cmd ' 2024-04-20T20:19:02.3287096Z +++ extract_trap_cmd 2024-04-20T20:19:02.3287537Z +++ printf '%s\n' '' 2024-04-20T20:19:02.3288439Z ++ printf '%s\n' cleanup_workspace 2024-04-20T20:19:02.3292572Z + trap -- ' 2024-04-20T20:19:02.3293356Z cleanup_workspace' EXIT 2024-04-20T20:19:02.3294122Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-04-20T20:19:02.8890052Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-04-20T20:19:02.8921889Z + echo 'Environment variables:' 2024-04-20T20:19:02.8922662Z Environment variables: 2024-04-20T20:19:02.8923054Z + env 2024-04-20T20:19:02.8932710Z INSTALLED_DB=yes 2024-04-20T20:19:02.8933774Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-04-20T20:19:02.8934629Z NVIDIA_VISIBLE_DEVICES=all 2024-04-20T20:19:02.8935373Z NV_NVML_DEV_VERSION=12.1.105-1 2024-04-20T20:19:02.8936780Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2024-04-20T20:19:02.8937544Z CONTINUE_THROUGH_ERROR=False 2024-04-20T20:19:02.8938157Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-04-20T20:19:02.8938864Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-04-20T20:19:02.8939580Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm80 2024-04-20T20:19:02.8940503Z HOSTNAME=3755bfd73a94 2024-04-20T20:19:02.8941879Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.8942936Z GITHUB_ACTION=__self 2024-04-20T20:19:02.8943359Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-20T20:19:02.8948386Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-04-20T20:19:02.8953647Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-04-20T20:19:02.8954277Z NV_NVTX_VERSION=12.1.105-1 2024-04-20T20:19:02.8954683Z GITHUB_RUN_NUMBER=3976 2024-04-20T20:19:02.8955114Z TEST_CONFIG=inductor_huggingface_perf 2024-04-20T20:19:02.8955620Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-20T20:19:02.8956190Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-20T20:19:02.8956745Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-04-20T20:19:02.8957278Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-04-20T20:19:02.8957789Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-04-20T20:19:02.8958252Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-04-20T20:19:02.8958845Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-20T20:19:02.8959414Z GITHUB_REF_TYPE=branch 2024-04-20T20:19:02.8959810Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-20T20:19:02.8960244Z NCCL_VERSION=2.17.1-1 2024-04-20T20:19:02.8960697Z BASE_SHA=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.8961270Z XLA_CUDA= 2024-04-20T20:19:02.8962824Z HUGGING_FACE_HUB_TOKEN=*** 2024-04-20T20:19:02.8963961Z *** 2024-04-20T20:19:02.8964285Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-20T20:19:02.8965031Z GITHUB_REPOSITORY_ID=65600975 2024-04-20T20:19:02.8965472Z GITHUB_ACTIONS=true 2024-04-20T20:19:02.8965853Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:19:02.8966491Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-04-20T20:19:02.8967152Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-04-20T20:19:02.8967720Z SHA1=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.8968331Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-20T20:19:02.8968916Z GITHUB_SHA=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.8970063Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main 2024-04-20T20:19:02.8971033Z UCC_HOME=/usr 2024-04-20T20:19:02.8971418Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-04-20T20:19:02.8971910Z VERBOSE_TEST_LOGS=False 2024-04-20T20:19:02.8972313Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-20T20:19:02.8973031Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-04-20T20:19:02.8973702Z GITHUB_REF=refs/heads/main 2024-04-20T20:19:02.8974188Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-04-20T20:19:02.8974634Z SHARD_NUMBER=1 2024-04-20T20:19:02.8974978Z GITHUB_REF_PROTECTED=true 2024-04-20T20:19:02.8975383Z HOME=/var/lib/jenkins 2024-04-20T20:19:02.8975813Z GITHUB_API_URL=https://api.github.com 2024-04-20T20:19:02.8976338Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-20T20:19:02.8976895Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-20T20:19:02.8977636Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf-nightly 2024-04-20T20:19:02.8978190Z CUDA_VERSION=12.1.1 2024-04-20T20:19:02.8978682Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-04-20T20:19:02.8979212Z NUM_TEST_SHARDS=3 2024-04-20T20:19:02.8979556Z UCX_HOME=/usr 2024-04-20T20:19:02.8980184Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-04-20T20:19:02.8981853Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.8983477Z JOB_NAME=cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:19:02.8985062Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.8986567Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-20T20:19:02.8987411Z GITHUB_EVENT_NAME=schedule 2024-04-20T20:19:02.8989134Z DASHBOARD_TAG=training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true 2024-04-20T20:19:02.8990762Z GITHUB_RUN_ID=8763335203 2024-04-20T20:19:02.8991356Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-04-20T20:19:02.8992019Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-04-20T20:19:02.8993332Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.8994469Z GITHUB_ACTOR=pytorchmergebot 2024-04-20T20:19:02.8994962Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-04-20T20:19:02.8995410Z PR_NUMBER= 2024-04-20T20:19:02.8995725Z GITHUB_RUN_ATTEMPT=1 2024-04-20T20:19:02.8996115Z ANACONDA_PYTHON_VERSION=3.10 2024-04-20T20:19:02.8996645Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-20T20:19:02.8997186Z TERM=xterm 2024-04-20T20:19:02.8997582Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-04-20T20:19:02.8998075Z INSTALLED_VISION=yes 2024-04-20T20:19:02.8998440Z BRANCH=main 2024-04-20T20:19:02.8998770Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-20T20:19:02.8999244Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-20T20:19:02.8999746Z CUDA_PATH=/usr/local/cuda 2024-04-20T20:19:02.9000718Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-20T20:19:02.9001726Z GITHUB_SERVER_URL=https://github.com 2024-04-20T20:19:02.9002315Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-20T20:19:02.9002888Z REENABLED_ISSUES= 2024-04-20T20:19:02.9003218Z SHLVL=1 2024-04-20T20:19:02.9003679Z MAX_JOBS=10 2024-04-20T20:19:02.9004058Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-04-20T20:19:02.9004482Z NVARCH=x86_64 2024-04-20T20:19:02.9004815Z GITHUB_ACTOR_ID=97764156 2024-04-20T20:19:02.9005358Z GITHUB_WORKFLOW_SHA=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.9005976Z GITHUB_REF_NAME=main 2024-04-20T20:19:02.9006448Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-04-20T20:19:02.9007279Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-20T20:19:02.9007994Z GITHUB_JOB=test 2024-04-20T20:19:02.9008473Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-04-20T20:19:02.9009135Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-20T20:19:02.9009754Z NO_TEST_TIMEOUT=False 2024-04-20T20:19:02.9010133Z TD_DISTRIBUTED=False 2024-04-20T20:19:02.9010589Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-04-20T20:19:02.9011139Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-20T20:19:02.9011950Z NV_NVPROF_VERSION=12.1.105-1 2024-04-20T20:19:02.9012512Z GITHUB_RETENTION_DAYS=90 2024-04-20T20:19:02.9013265Z OPENSSL_DIR=/opt/openssl 2024-04-20T20:19:02.9013882Z GITHUB_ACTION_REPOSITORY= 2024-04-20T20:19:02.9015317Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-20T20:19:02.9016807Z GITHUB_BASE_REF= 2024-04-20T20:19:02.9017257Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-20T20:19:02.9017801Z CI=true 2024-04-20T20:19:02.9018338Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-04-20T20:19:02.9018909Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-20T20:19:02.9019473Z JOB_ID=24052568002 2024-04-20T20:19:02.9019970Z INSTALLED_PROTOBUF=yes 2024-04-20T20:19:02.9020473Z GITHUB_HEAD_REF= 2024-04-20T20:19:02.9020884Z GITHUB_ACTION_REF= 2024-04-20T20:19:02.9021850Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-20T20:19:02.9047488Z GITHUB_WORKFLOW=inductor-A100-perf-nightly 2024-04-20T20:19:02.9048189Z DEBIAN_FRONTEND=noninteractive 2024-04-20T20:19:02.9049513Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.9050567Z NO_TD=False 2024-04-20T20:19:02.9050923Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-20T20:19:02.9051422Z _=/usr/bin/env 2024-04-20T20:19:02.9052028Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-04-20T20:19:02.9147128Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-04-20T20:19:02.9148670Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-04-20T20:19:02.9150403Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-04-20T20:19:02.9151969Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-04-20T20:19:02.9152746Z + BUILD_DIR=build 2024-04-20T20:19:02.9153117Z + BUILD_RENAMED_DIR=build_renamed 2024-04-20T20:19:02.9153588Z + BUILD_BIN_DIR=build/bin 2024-04-20T20:19:02.9153996Z + SHARD_NUMBER=1 2024-04-20T20:19:02.9154346Z + NUM_TEST_SHARDS=3 2024-04-20T20:19:02.9154720Z + export VALGRIND=ON 2024-04-20T20:19:02.9155091Z + VALGRIND=ON 2024-04-20T20:19:02.9155665Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *clang9* ]] 2024-04-20T20:19:02.9156397Z + [[ 0 == \1 ]] 2024-04-20T20:19:02.9156742Z + [[ False == \1 ]] 2024-04-20T20:19:02.9157341Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *bazel* ]] 2024-04-20T20:19:02.9157987Z ++ realpath build/custom_test_artifacts 2024-04-20T20:19:02.9167223Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-04-20T20:19:02.9168565Z + [[ -n '' ]] 2024-04-20T20:19:02.9169054Z + echo 'Environment variables' 2024-04-20T20:19:02.9169511Z Environment variables 2024-04-20T20:19:02.9169866Z + env 2024-04-20T20:19:02.9178356Z INSTALLED_DB=yes 2024-04-20T20:19:02.9179160Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-04-20T20:19:02.9179841Z NVIDIA_VISIBLE_DEVICES=all 2024-04-20T20:19:02.9180591Z NV_NVML_DEV_VERSION=12.1.105-1 2024-04-20T20:19:02.9182281Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch 2024-04-20T20:19:02.9183661Z CONTINUE_THROUGH_ERROR=False 2024-04-20T20:19:02.9184801Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-04-20T20:19:02.9185905Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-04-20T20:19:02.9187270Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm80 2024-04-20T20:19:02.9188419Z HOSTNAME=3755bfd73a94 2024-04-20T20:19:02.9190484Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.9192458Z GITHUB_ACTION=__self 2024-04-20T20:19:02.9193257Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-04-20T20:19:02.9200532Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-04-20T20:19:02.9205793Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-04-20T20:19:02.9206479Z NV_NVTX_VERSION=12.1.105-1 2024-04-20T20:19:02.9206898Z GITHUB_RUN_NUMBER=3976 2024-04-20T20:19:02.9207311Z TEST_CONFIG=inductor_huggingface_perf 2024-04-20T20:19:02.9207828Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-04-20T20:19:02.9208676Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-04-20T20:19:02.9209235Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-04-20T20:19:02.9209750Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-04-20T20:19:02.9210269Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-04-20T20:19:02.9210750Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-04-20T20:19:02.9211349Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-04-20T20:19:02.9211904Z GITHUB_REF_TYPE=branch 2024-04-20T20:19:02.9212306Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-04-20T20:19:02.9212754Z NCCL_VERSION=2.17.1-1 2024-04-20T20:19:02.9213527Z BASE_SHA=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.9214065Z XLA_CUDA= 2024-04-20T20:19:02.9214692Z HUGGING_FACE_HUB_TOKEN=*** 2024-04-20T20:19:02.9215164Z *** 2024-04-20T20:19:02.9215490Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-04-20T20:19:02.9215948Z GITHUB_REPOSITORY_ID=65600975 2024-04-20T20:19:02.9216438Z GITHUB_ACTIONS=true 2024-04-20T20:19:02.9216829Z NVIDIA_DRIVER_CAPABILITIES=all 2024-04-20T20:19:02.9217452Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-04-20T20:19:02.9218113Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-04-20T20:19:02.9218784Z SHA1=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.9219524Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-04-20T20:19:02.9220093Z GITHUB_SHA=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.9221232Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main 2024-04-20T20:19:02.9222160Z UCC_HOME=/usr 2024-04-20T20:19:02.9222552Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-04-20T20:19:02.9223016Z VERBOSE_TEST_LOGS=False 2024-04-20T20:19:02.9223413Z NVIDIA_PRODUCT_NAME=CUDA 2024-04-20T20:19:02.9223975Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-04-20T20:19:02.9224537Z GITHUB_REF=refs/heads/main 2024-04-20T20:19:02.9224979Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-04-20T20:19:02.9225551Z SHARD_NUMBER=1 2024-04-20T20:19:02.9225907Z GITHUB_REF_PROTECTED=true 2024-04-20T20:19:02.9226404Z HOME=/var/lib/jenkins 2024-04-20T20:19:02.9226828Z GITHUB_API_URL=https://api.github.com 2024-04-20T20:19:02.9227355Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-04-20T20:19:02.9227933Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af 2024-04-20T20:19:02.9228885Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf-nightly 2024-04-20T20:19:02.9229434Z CUDA_VERSION=12.1.1 2024-04-20T20:19:02.9229952Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-04-20T20:19:02.9230481Z NUM_TEST_SHARDS=3 2024-04-20T20:19:02.9230819Z UCX_HOME=/usr 2024-04-20T20:19:02.9231456Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-04-20T20:19:02.9232881Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.9234469Z JOB_NAME=cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 1, 3, linux.gcp.a100.large) 2024-04-20T20:19:02.9236093Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.9237576Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json 2024-04-20T20:19:02.9238420Z GITHUB_EVENT_NAME=schedule 2024-04-20T20:19:02.9240165Z DASHBOARD_TAG=training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true 2024-04-20T20:19:02.9241786Z GITHUB_RUN_ID=8763335203 2024-04-20T20:19:02.9242330Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-04-20T20:19:02.9242993Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-04-20T20:19:02.9244303Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.9245446Z GITHUB_ACTOR=pytorchmergebot 2024-04-20T20:19:02.9245918Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-04-20T20:19:02.9246576Z PR_NUMBER= 2024-04-20T20:19:02.9246914Z GITHUB_RUN_ATTEMPT=1 2024-04-20T20:19:02.9247271Z VALGRIND=ON 2024-04-20T20:19:02.9247791Z ANACONDA_PYTHON_VERSION=3.10 2024-04-20T20:19:02.9248333Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-04-20T20:19:02.9248896Z TERM=xterm 2024-04-20T20:19:02.9249303Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-04-20T20:19:02.9249810Z INSTALLED_VISION=yes 2024-04-20T20:19:02.9250185Z BRANCH=main 2024-04-20T20:19:02.9250687Z OPENSSL_ROOT_DIR=/opt/openssl 2024-04-20T20:19:02.9251571Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-04-20T20:19:02.9252550Z CUDA_PATH=/usr/local/cuda 2024-04-20T20:19:02.9254165Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-04-20T20:19:02.9255141Z GITHUB_SERVER_URL=https://github.com 2024-04-20T20:19:02.9255723Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea 2024-04-20T20:19:02.9256347Z REENABLED_ISSUES= 2024-04-20T20:19:02.9256694Z SHLVL=1 2024-04-20T20:19:02.9256979Z MAX_JOBS=10 2024-04-20T20:19:02.9257355Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-04-20T20:19:02.9257770Z NVARCH=x86_64 2024-04-20T20:19:02.9258126Z GITHUB_ACTOR_ID=97764156 2024-04-20T20:19:02.9258656Z GITHUB_WORKFLOW_SHA=f20e3ae0c36146c962a5665018e9ad662a7cf211 2024-04-20T20:19:02.9259278Z GITHUB_REF_NAME=main 2024-04-20T20:19:02.9259767Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-04-20T20:19:02.9260601Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-04-20T20:19:02.9261296Z GITHUB_JOB=test 2024-04-20T20:19:02.9261780Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-04-20T20:19:02.9262500Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-04-20T20:19:02.9263125Z NO_TEST_TIMEOUT=False 2024-04-20T20:19:02.9263496Z TD_DISTRIBUTED=False 2024-04-20T20:19:02.9263943Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-04-20T20:19:02.9264472Z GITHUB_REPOSITORY=pytorch/pytorch 2024-04-20T20:19:02.9264973Z NV_NVPROF_VERSION=12.1.105-1 2024-04-20T20:19:02.9265386Z GITHUB_RETENTION_DAYS=90 2024-04-20T20:19:02.9265793Z OPENSSL_DIR=/opt/openssl 2024-04-20T20:19:02.9266244Z GITHUB_ACTION_REPOSITORY= 2024-04-20T20:19:02.9267563Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-20T20:19:02.9268923Z GITHUB_BASE_REF= 2024-04-20T20:19:02.9269510Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-04-20T20:19:02.9269960Z CI=true 2024-04-20T20:19:02.9270321Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-04-20T20:19:02.9270818Z GITHUB_REPOSITORY_OWNER=pytorch 2024-04-20T20:19:02.9271265Z JOB_ID=24052568002 2024-04-20T20:19:02.9271634Z INSTALLED_PROTOBUF=yes 2024-04-20T20:19:02.9272010Z GITHUB_HEAD_REF= 2024-04-20T20:19:02.9272364Z GITHUB_ACTION_REF= 2024-04-20T20:19:02.9272901Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-04-20T20:19:02.9273584Z GITHUB_WORKFLOW=inductor-A100-perf-nightly 2024-04-20T20:19:02.9274129Z DEBIAN_FRONTEND=noninteractive 2024-04-20T20:19:02.9275317Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_37621534-ec1a-4d65-b13f-694319a5b841 2024-04-20T20:19:02.9276417Z NO_TD=False 2024-04-20T20:19:02.9276750Z SKIP_SCCACHE_INITIALIZATION=1 2024-04-20T20:19:02.9277184Z _=/usr/bin/env 2024-04-20T20:19:02.9277588Z + echo 'Testing pytorch' 2024-04-20T20:19:02.9277989Z Testing pytorch 2024-04-20T20:19:02.9278364Z + export LANG=C.UTF-8 2024-04-20T20:19:02.9278762Z + LANG=C.UTF-8 2024-04-20T20:19:02.9279099Z + PR_NUMBER= 2024-04-20T20:19:02.9279507Z + [[ inductor_huggingface_perf == \d\e\f\a\u\l\t ]] 2024-04-20T20:19:02.9280194Z + [[ inductor_huggingface_perf == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-04-20T20:19:02.9280856Z + [[ inductor_huggingface_perf == \s\l\o\w ]] 2024-04-20T20:19:02.9281671Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *slow-gradcheck* ]] 2024-04-20T20:19:02.9282525Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *cuda* ]] 2024-04-20T20:19:02.9283221Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-20T20:19:02.9283793Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-04-20T20:19:02.9284353Z + [[ inductor_huggingface_perf == *crossref* ]] 2024-04-20T20:19:02.9285217Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *rocm* ]] 2024-04-20T20:19:02.9286014Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *xpu* ]] 2024-04-20T20:19:02.9286870Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *-bazel-* ]] 2024-04-20T20:19:02.9287562Z + pip_install --user ninja==1.10.2 2024-04-20T20:19:02.9288206Z + pip install --progress-bar off --user ninja==1.10.2 2024-04-20T20:19:03.4613040Z Collecting ninja==1.10.2 2024-04-20T20:19:03.6021095Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-04-20T20:19:03.6550711Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-04-20T20:19:05.2393272Z Installing collected packages: ninja 2024-04-20T20:19:05.2489418Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-04-20T20:19:05.2491123Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-04-20T20:19:05.2547229Z Successfully installed ninja-1.10.2 2024-04-20T20:19:05.3574466Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-20T20:19:05.3577263Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-20T20:19:05.3578851Z + install_tlparse 2024-04-20T20:19:05.3579515Z + pip_install --user tlparse==0.3.7 2024-04-20T20:19:05.3580204Z + pip install --progress-bar off --user tlparse==0.3.7 2024-04-20T20:19:05.9027663Z Collecting tlparse==0.3.7 2024-04-20T20:19:06.0508938Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-04-20T20:19:06.1022334Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-04-20T20:19:07.8470102Z Installing collected packages: tlparse 2024-04-20T20:19:07.8976146Z Successfully installed tlparse-0.3.7 2024-04-20T20:19:07.9921581Z ++ python -m site --user-base 2024-04-20T20:19:08.0145658Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-04-20T20:19:08.0148659Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *asan* ]] 2024-04-20T20:19:08.0149493Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *-debug* ]] 2024-04-20T20:19:08.0150329Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *-bazel-* ]] 2024-04-20T20:19:08.0151524Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm80. Expect the assertion to pass' 2024-04-20T20:19:08.0153066Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm80. Expect the assertion to pass 2024-04-20T20:19:08.0153969Z + cd test 2024-04-20T20:19:08.0154625Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-04-20T20:19:09.9119153Z + [[ inductor_huggingface_perf == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-04-20T20:19:09.9120050Z + [[ inductor_huggingface_perf == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-04-20T20:19:09.9120712Z + DYNAMO_BENCHMARK_FLAGS=() 2024-04-20T20:19:09.9121236Z + [[ inductor_huggingface_perf == *dynamo_eager* ]] 2024-04-20T20:19:09.9122061Z + [[ inductor_huggingface_perf == *aot_eager* ]] 2024-04-20T20:19:09.9122690Z + [[ inductor_huggingface_perf == *aot_inductor* ]] 2024-04-20T20:19:09.9123322Z + [[ inductor_huggingface_perf == *inductor* ]] 2024-04-20T20:19:09.9123914Z + [[ inductor_huggingface_perf != *perf* ]] 2024-04-20T20:19:09.9124483Z + [[ inductor_huggingface_perf == *dynamic* ]] 2024-04-20T20:19:09.9125093Z + [[ inductor_huggingface_perf == *cpu_inductor* ]] 2024-04-20T20:19:09.9126230Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-04-20T20:19:09.9135881Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *tbb* ]] 2024-04-20T20:19:09.9153057Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *libtorch* ]] 2024-04-20T20:19:09.9153926Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *-bazel-* ]] 2024-04-20T20:19:09.9156215Z + cd test 2024-04-20T20:19:09.9156871Z + python -c 'import torch; print(torch.__config__.show())' 2024-04-20T20:19:11.4909933Z PyTorch built with: 2024-04-20T20:19:11.4910926Z - GCC 9.4 2024-04-20T20:19:11.4911309Z - C++ Version: 201703 2024-04-20T20:19:11.4912408Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-20T20:19:11.4913783Z - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-20T20:19:11.4914615Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-20T20:19:11.4915259Z - LAPACK is enabled (usually provided by MKL) 2024-04-20T20:19:11.4915849Z - NNPACK is enabled 2024-04-20T20:19:11.4916399Z - CPU capability usage: AVX512 2024-04-20T20:19:11.4916941Z - CUDA Runtime 12.1 2024-04-20T20:19:11.4917573Z - NVCC architecture flags: -gencode;arch=compute_80,code=sm_80 2024-04-20T20:19:11.4918259Z - CuDNN 8.9.2 2024-04-20T20:19:11.4918643Z - Magma 2.6.1 2024-04-20T20:19:11.4929218Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DTMP_LIBKINETO_NANOSECOND -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-04-20T20:19:11.4938725Z 2024-04-20T20:19:11.8248266Z + cd test 2024-04-20T20:19:11.8249232Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-04-20T20:19:13.3190973Z ATen/Parallel: 2024-04-20T20:19:13.3192201Z at::get_num_threads() : 6 2024-04-20T20:19:13.3192677Z at::get_num_interop_threads() : 12 2024-04-20T20:19:13.3193183Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-04-20T20:19:13.3193664Z omp_get_max_threads() : 6 2024-04-20T20:19:13.3195006Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-04-20T20:19:13.3196049Z mkl_get_max_threads() : 6 2024-04-20T20:19:13.3196904Z Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361) 2024-04-20T20:19:13.3197687Z std::thread::hardware_concurrency() : 12 2024-04-20T20:19:13.3198195Z Environment variables: 2024-04-20T20:19:13.3198614Z OMP_NUM_THREADS : [not set] 2024-04-20T20:19:13.3199060Z MKL_NUM_THREADS : [not set] 2024-04-20T20:19:13.3199510Z ATen parallel backend: OpenMP 2024-04-20T20:19:13.3199814Z 2024-04-20T20:19:13.5789179Z + [[ inductor_huggingface_perf == *backward* ]] 2024-04-20T20:19:13.5789946Z + [[ inductor_huggingface_perf == *xla* ]] 2024-04-20T20:19:13.5790550Z + [[ inductor_huggingface_perf == *executorch* ]] 2024-04-20T20:19:13.5791352Z + [[ inductor_huggingface_perf == \j\i\t\_\l\e\g\a\c\y ]] 2024-04-20T20:19:13.5792422Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *libtorch* ]] 2024-04-20T20:19:13.5793937Z + [[ inductor_huggingface_perf == distributed ]] 2024-04-20T20:19:13.5794556Z + [[ inductor_huggingface_perf == deploy ]] 2024-04-20T20:19:13.5795583Z + [[ inductor_huggingface_perf == *inductor_distributed* ]] 2024-04-20T20:19:13.5796281Z + [[ inductor_huggingface_perf == *huggingface* ]] 2024-04-20T20:19:13.5796850Z + install_torchvision 2024-04-20T20:19:13.5797242Z + local orig_preload 2024-04-20T20:19:13.5797620Z + local commit 2024-04-20T20:19:13.5798019Z ++ get_pinned_commit vision 2024-04-20T20:19:13.5798493Z ++ cat .github/ci_commit_pins/vision.txt 2024-04-20T20:19:13.5817339Z + commit=2c4665ffbb64f03f5d18016d3398af4ac4da5f03 2024-04-20T20:19:13.5818231Z + orig_preload= 2024-04-20T20:19:13.5818749Z + '[' -n '' ']' 2024-04-20T20:19:13.5819856Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@2c4665ffbb64f03f5d18016d3398af4ac4da5f03 2024-04-20T20:19:13.5821761Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@2c4665ffbb64f03f5d18016d3398af4ac4da5f03 2024-04-20T20:19:14.0386007Z Collecting git+https://github.com/pytorch/vision.git@2c4665ffbb64f03f5d18016d3398af4ac4da5f03 2024-04-20T20:19:14.0390765Z Cloning https://github.com/pytorch/vision.git (to revision 2c4665ffbb64f03f5d18016d3398af4ac4da5f03) to /tmp/pip-req-build-9xtdrw6_ 2024-04-20T20:19:14.0420616Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-9xtdrw6_ 2024-04-20T20:19:16.2600365Z Running command git rev-parse -q --verify 'sha^2c4665ffbb64f03f5d18016d3398af4ac4da5f03' 2024-04-20T20:19:16.2627685Z Running command git fetch -q https://github.com/pytorch/vision.git 2c4665ffbb64f03f5d18016d3398af4ac4da5f03 2024-04-20T20:19:18.0123723Z Running command git checkout -q 2c4665ffbb64f03f5d18016d3398af4ac4da5f03 2024-04-20T20:19:18.4866073Z Resolved https://github.com/pytorch/vision.git to commit 2c4665ffbb64f03f5d18016d3398af4ac4da5f03 2024-04-20T20:19:21.0922639Z Preparing metadata (setup.py) ... [?25l- \ done 2024-04-20T20:19:21.0981418Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+2c4665f) (1.21.2) 2024-04-20T20:19:21.0986004Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+2c4665f) (2.4.0a0+gitf20e3ae) 2024-04-20T20:19:21.0992851Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+2c4665f) (10.2.0) 2024-04-20T20:19:21.1247016Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+2c4665f) (3.13.1) 2024-04-20T20:19:21.1258292Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+2c4665f) (4.11.0) 2024-04-20T20:19:21.1262713Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+2c4665f) (1.12) 2024-04-20T20:19:21.1267005Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+2c4665f) (2.8.8) 2024-04-20T20:19:21.1271149Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+2c4665f) (3.1.3) 2024-04-20T20:19:21.1275470Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+2c4665f) (2024.2.0) 2024-04-20T20:19:21.1797489Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+2c4665f) (2.1.5) 2024-04-20T20:19:21.1988455Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+2c4665f) (1.3.0) 2024-04-20T20:19:21.2082101Z Building wheels for collected packages: torchvision 2024-04-20T20:21:11.2717759Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-04-20T20:21:11.2789895Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+2c4665f-cp310-cp310-linux_x86_64.whl size=2102630 sha256=f5a720a40e4eef1cc6639d5e4d2c331429792f3eea159530e858cab9c42b6364 2024-04-20T20:21:11.2793269Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/fe/8e/eb/30dab9378a6be22c0f3a178e38ae735f03d8debd545032e10c 2024-04-20T20:21:11.2839499Z Successfully built torchvision 2024-04-20T20:21:12.7044033Z Installing collected packages: torchvision 2024-04-20T20:21:13.2185758Z Successfully installed torchvision-0.19.0a0+2c4665f 2024-04-20T20:21:13.3557891Z + '[' -n '' ']' 2024-04-20T20:21:13.3558535Z + id=0 2024-04-20T20:21:13.3559126Z + test_dynamo_benchmark huggingface 0 2024-04-20T20:21:13.3568520Z ++ pwd 2024-04-20T20:21:13.3573269Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-04-20T20:21:13.3574084Z + local suite=huggingface 2024-04-20T20:21:13.3574479Z + shift 2024-04-20T20:21:13.3574816Z + local shard_id=0 2024-04-20T20:21:13.3575578Z + shift 2024-04-20T20:21:13.3575981Z + [[ inductor_huggingface_perf == *perf_compare* ]] 2024-04-20T20:21:13.3576610Z + [[ inductor_huggingface_perf == *perf* ]] 2024-04-20T20:21:13.3577352Z + test_single_dynamo_benchmark dashboard huggingface 0 2024-04-20T20:21:13.3580177Z ++ pwd 2024-04-20T20:21:13.3583085Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-04-20T20:21:13.3583965Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-04-20T20:21:13.3605293Z + local name=dashboard 2024-04-20T20:21:13.3606023Z + shift 2024-04-20T20:21:13.3606621Z + local suite=huggingface 2024-04-20T20:21:13.3607024Z + shift 2024-04-20T20:21:13.3607336Z + local shard_id=0 2024-04-20T20:21:13.3607713Z + shift 2024-04-20T20:21:13.3608029Z + partition_flags=() 2024-04-20T20:21:13.3608410Z + local partition_flags 2024-04-20T20:21:13.3609084Z + [[ -n 3 ]] 2024-04-20T20:21:13.3609482Z + [[ -n 0 ]] 2024-04-20T20:21:13.3610842Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-04-20T20:21:13.3611734Z + [[ inductor_huggingface_perf == *perf_compare* ]] 2024-04-20T20:21:13.3612574Z + [[ inductor_huggingface_perf == *perf* ]] 2024-04-20T20:21:13.3613905Z + test_perf_for_dashboard huggingface --device cuda --total-partitions 3 --partition-id 0 2024-04-20T20:21:13.3615074Z ++ pwd 2024-04-20T20:21:13.3615648Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-04-20T20:21:13.3616479Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-04-20T20:21:13.3631238Z + local suite=huggingface 2024-04-20T20:21:13.3631942Z + shift 2024-04-20T20:21:13.3632447Z + local backend=inductor 2024-04-20T20:21:13.3633215Z + modes=() 2024-04-20T20:21:13.3633805Z + local modes 2024-04-20T20:21:13.3635746Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *training-true* ]] 2024-04-20T20:21:13.3637461Z + modes+=(training) 2024-04-20T20:21:13.3639251Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *inference-true* ]] 2024-04-20T20:21:13.3641018Z + modes+=(inference) 2024-04-20T20:21:13.3641429Z + targets=(accuracy performance) 2024-04-20T20:21:13.3641870Z + local targets 2024-04-20T20:21:13.3642250Z + for mode in "${modes[@]}" 2024-04-20T20:21:13.3642702Z + [[ training == \i\n\f\e\r\e\n\c\e ]] 2024-04-20T20:21:13.3643211Z + [[ training == \t\r\a\i\n\i\n\g ]] 2024-04-20T20:21:13.3643663Z + dtype=amp 2024-04-20T20:21:13.3644009Z + for target in "${targets[@]}" 2024-04-20T20:21:13.3644528Z + target_flag=("--${target}") 2024-04-20T20:21:13.3644961Z + local target_flag 2024-04-20T20:21:13.3645690Z + [[ accuracy == \p\e\r\f\o\r\m\a\n\c\e ]] 2024-04-20T20:21:13.3646225Z + [[ accuracy == \a\c\c\u\r\a\c\y ]] 2024-04-20T20:21:13.3646854Z + target_flag+=(--no-translation-validation) 2024-04-20T20:21:13.3648788Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *default-true* ]] 2024-04-20T20:21:13.3652724Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_amp_training_cuda_accuracy.csv 2024-04-20T20:21:17.0742905Z 2024-04-20T20:21:17.0744805Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-04-20T20:23:14.4499641Z pass 2024-04-20T20:23:20.4070025Z 2024-04-20T20:23:24.7247633Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:23:24.7248721Z loading model: 0it [00:04, ?it/s] 2024-04-20T20:23:24.7252569Z cuda train AllenaiLongformerBase 2024-04-20T20:23:26.8864763Z W0420 20:23:26.885000 140019058332864 torch/_dynamo/variables/tensor.py:687] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-04-20T20:23:26.8866532Z W0420 20:23:26.885000 140019058332864 torch/_dynamo/variables/tensor.py:687] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-04-20T20:23:26.8867951Z W0420 20:23:26.885000 140019058332864 torch/_dynamo/variables/tensor.py:687] [3/0] or: 2024-04-20T20:23:26.8869304Z W0420 20:23:26.885000 140019058332864 torch/_dynamo/variables/tensor.py:687] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-04-20T20:23:26.8870946Z W0420 20:23:26.885000 140019058332864 torch/_dynamo/variables/tensor.py:687] [3/0] to include these operations in the captured graph. 2024-04-20T20:23:26.8872395Z W0420 20:23:26.885000 140019058332864 torch/_dynamo/variables/tensor.py:687] [3/0] 2024-04-20T20:25:11.4743202Z W0420 20:25:11.472000 140019058332864 torch/_logging/_internal.py:1016] [16/0] Profiler function will be ignored 2024-04-20T20:25:55.9065318Z pass 2024-04-20T20:26:03.9162088Z 2024-04-20T20:26:09.7738752Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:26:09.7739328Z loading model: 0it [00:05, ?it/s] 2024-04-20T20:26:09.7740075Z cuda train BartForCausalLM 2024-04-20T20:26:48.9455314Z pass 2024-04-20T20:26:54.4987142Z 2024-04-20T20:27:04.6212231Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:27:04.6212811Z loading model: 0it [00:10, ?it/s] 2024-04-20T20:27:04.6213520Z cuda train BartForConditionalGeneration 2024-04-20T20:28:25.1779810Z pass 2024-04-20T20:28:30.7129072Z 2024-04-20T20:28:33.8270055Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:28:33.8270985Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:28:33.8271586Z cuda train BertForMaskedLM 2024-04-20T20:29:16.1256140Z W0420 20:29:16.123000 140648088315072 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:29:57.0665099Z pass 2024-04-20T20:30:04.3436075Z 2024-04-20T20:30:07.1380935Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:30:07.1381486Z loading model: 0it [00:02, ?it/s] 2024-04-20T20:30:07.1382226Z cuda train BertForQuestionAnswering 2024-04-20T20:30:47.6678655Z W0420 20:30:47.665000 140646168356032 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:31:28.6892160Z pass 2024-04-20T20:31:36.3619239Z 2024-04-20T20:32:16.5325097Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:32:16.5325686Z loading model: 0it [00:40, ?it/s] 2024-04-20T20:32:16.5326290Z cuda train BlenderbotForCausalLM 2024-04-20T20:32:16.5326864Z pass_due_to_skip 2024-04-20T20:32:21.2821342Z 2024-04-20T20:32:23.8282200Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:32:23.8284229Z loading model: 0it [00:02, ?it/s] 2024-04-20T20:32:23.8285235Z cuda train BlenderbotSmallForCausalLM 2024-04-20T20:32:23.8552381Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-04-20T20:32:57.5896523Z pass 2024-04-20T20:33:02.9659119Z 2024-04-20T20:33:05.9091934Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:33:05.9093067Z loading model: 0it [00:02, ?it/s] 2024-04-20T20:33:05.9093688Z cuda train BlenderbotSmallForConditionalGeneration 2024-04-20T20:33:05.9684191Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-04-20T20:34:07.7494056Z pass 2024-04-20T20:34:13.3287052Z 2024-04-20T20:34:16.7029750Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:34:16.7030838Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:34:16.7031434Z cuda train CamemBert 2024-04-20T20:34:58.5480540Z W0420 20:34:58.546000 140503730947264 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:35:39.3255051Z pass 2024-04-20T20:35:46.5323585Z 2024-04-20T20:35:50.3301292Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:35:50.3301862Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:35:50.3302404Z cuda train DebertaForMaskedLM 2024-04-20T20:36:36.8433166Z W0420 20:36:36.841000 139941540627648 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:37:14.2419887Z pass 2024-04-20T20:37:21.0575434Z 2024-04-20T20:37:24.3018089Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:37:24.3018670Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:37:24.3019237Z cuda train DebertaForQuestionAnswering 2024-04-20T20:38:08.6555249Z W0420 20:38:08.653000 139822865880256 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:38:41.5069160Z E0420 20:38:41.505000 139822865880256 torch/_dynamo/utils.py:1440] RMSE (res-fp64): 0.01709, (ref-fp64): 0.00506 and shape=torch.Size([2]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000 2024-04-20T20:38:41.5071539Z E0420 20:38:41.505000 139822865880256 torch/_dynamo/utils.py:1354] Accuracy failed for key name qa_outputs.bias 2024-04-20T20:38:41.5122672Z fail_accuracy 2024-04-20T20:38:48.6388147Z 2024-04-20T20:39:06.5385003Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:39:06.5385592Z loading model: 0it [00:17, ?it/s] 2024-04-20T20:39:06.5386126Z cuda train DebertaV2ForMaskedLM 2024-04-20T20:39:06.5387145Z pass_due_to_skip 2024-04-20T20:39:11.1891886Z 2024-04-20T20:39:26.1692684Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:39:26.1693604Z loading model: 0it [00:14, ?it/s] 2024-04-20T20:39:26.1694372Z cuda train DebertaV2ForQuestionAnswering 2024-04-20T20:40:47.2188571Z pass 2024-04-20T20:40:54.4370233Z 2024-04-20T20:40:56.9435176Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:40:56.9436094Z loading model: 0it [00:02, ?it/s] 2024-04-20T20:40:56.9437986Z cuda train DistilBertForMaskedLM 2024-04-20T20:41:23.0617879Z W0420 20:41:23.059000 140061743154368 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:41:46.8535223Z pass 2024-04-20T20:41:49.1963419Z accuracy pass_rate=81.25% 2024-04-20T20:41:49.1964474Z calls_captured gmean=0.00x mean=811.500x 2024-04-20T20:41:49.1968231Z unique_graphs gmean=0.00x mean=6.500x 2024-04-20T20:41:49.1969272Z graph_breaks gmean=0.00x mean=7.750x 2024-04-20T20:41:49.1970734Z unique_graph_breaks gmean=0.00x mean=4.812x 2024-04-20T20:41:49.1971360Z autograd_captures gmean=0.00x mean=0.000x 2024-04-20T20:41:49.1972116Z autograd_compiles gmean=0.00x mean=0.000x 2024-04-20T20:41:49.8213860Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *cudagraphs-true* ]] 2024-04-20T20:41:49.8217786Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --total-partitions 3 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_huggingface_amp_training_cuda_accuracy.csv 2024-04-20T20:41:57.8043428Z 2024-04-20T20:42:02.0520885Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:42:02.0521981Z loading model: 0it [00:04, ?it/s] 2024-04-20T20:42:02.0522810Z cuda train AlbertForMaskedLM 2024-04-20T20:42:40.5326115Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:42:40.5328996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T20:42:40.5331244Z pred = mod(**cloned_inputs) 2024-04-20T20:42:40.5338113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:42:40.5340065Z return forward_call(*args, **kwargs) 2024-04-20T20:42:40.5342643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2024-04-20T20:42:40.5344677Z outputs = self.albert( 2024-04-20T20:42:40.5346722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:42:40.5348742Z return forward_call(*args, **kwargs) 2024-04-20T20:42:40.5351402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-04-20T20:42:40.5353386Z embedding_output = self.embeddings( 2024-04-20T20:42:40.5355919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:42:40.5358089Z return forward_call(*args, **kwargs) 2024-04-20T20:42:40.5360916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-04-20T20:42:40.5363111Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:42:40.5363908Z 2024-04-20T20:42:41.6521040Z pass 2024-04-20T20:42:47.3290763Z 2024-04-20T20:42:51.6217709Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:42:51.6219100Z loading model: 0it [00:04, ?it/s] 2024-04-20T20:42:51.6219660Z cuda train AlbertForQuestionAnswering 2024-04-20T20:43:29.7831204Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:43:29.7833009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T20:43:29.7834261Z pred = mod(**cloned_inputs) 2024-04-20T20:43:29.7838685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:43:29.7839759Z return forward_call(*args, **kwargs) 2024-04-20T20:43:29.7841087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 1256, in forward 2024-04-20T20:43:29.7842291Z outputs = self.albert( 2024-04-20T20:43:29.7843442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:43:29.7844526Z return forward_call(*args, **kwargs) 2024-04-20T20:43:29.7845837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-04-20T20:43:29.7847022Z embedding_output = self.embeddings( 2024-04-20T20:43:29.7848197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:43:29.7849256Z return forward_call(*args, **kwargs) 2024-04-20T20:43:29.7850565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-04-20T20:43:29.7851831Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:43:29.7852258Z 2024-04-20T20:43:30.1452209Z W0420 20:43:30.144000 140657374348480 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:43:35.7313575Z pass 2024-04-20T20:43:41.1848149Z 2024-04-20T20:43:45.4896821Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:43:45.4897741Z loading model: 0it [00:04, ?it/s] 2024-04-20T20:43:45.4898288Z cuda train AllenaiLongformerBase 2024-04-20T20:43:47.6404650Z W0420 20:43:47.639000 140345960322240 torch/_dynamo/variables/tensor.py:687] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-04-20T20:43:47.6407837Z W0420 20:43:47.639000 140345960322240 torch/_dynamo/variables/tensor.py:687] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-04-20T20:43:47.6410157Z W0420 20:43:47.639000 140345960322240 torch/_dynamo/variables/tensor.py:687] [3/0] or: 2024-04-20T20:43:47.6412485Z W0420 20:43:47.639000 140345960322240 torch/_dynamo/variables/tensor.py:687] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-04-20T20:43:47.6415480Z W0420 20:43:47.639000 140345960322240 torch/_dynamo/variables/tensor.py:687] [3/0] to include these operations in the captured graph. 2024-04-20T20:43:47.6417782Z W0420 20:43:47.639000 140345960322240 torch/_dynamo/variables/tensor.py:687] [3/0] 2024-04-20T20:45:25.5518695Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:45:25.5525300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in torch_dynamo_resume_in_forward_at_1291 2024-04-20T20:45:25.5527068Z layer_outputs = layer_module( 2024-04-20T20:45:25.5528755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:45:25.5530589Z return forward_call(*args, **kwargs) 2024-04-20T20:45:25.5533410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-04-20T20:45:25.5535620Z self_attn_outputs = self.attention( 2024-04-20T20:45:25.5537801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:45:25.5540227Z return forward_call(*args, **kwargs) 2024-04-20T20:45:25.5542815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-04-20T20:45:25.5545028Z self_outputs = self.self( 2024-04-20T20:45:25.5547053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:45:25.5548937Z return forward_call(*args, **kwargs) 2024-04-20T20:45:25.5550780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 571, in forward 2024-04-20T20:45:25.5552523Z attn_scores = self._sliding_chunks_query_key_matmul( 2024-04-20T20:45:25.5554363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 836, in _sliding_chunks_query_key_matmul 2024-04-20T20:45:25.5556054Z query = self._chunk(query, window_overlap, getattr(self.config, "onnx_export", False)) 2024-04-20T20:45:25.5557926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 778, in _chunk 2024-04-20T20:45:25.5559363Z return hidden_states.as_strided(size=chunk_size, stride=chunk_stride) 2024-04-20T20:45:25.5559922Z 2024-04-20T20:45:25.9984129Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:45:25.9986127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1734, in forward 2024-04-20T20:45:25.9987461Z embedding_output = self.embeddings( 2024-04-20T20:45:25.9988652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:45:25.9989735Z return forward_call(*args, **kwargs) 2024-04-20T20:45:25.9991118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 470, in forward 2024-04-20T20:45:25.9992440Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:45:25.9992866Z 2024-04-20T20:45:26.0090150Z W0420 20:45:26.007000 140345960322240 torch/_logging/_internal.py:1016] [16/0] Profiler function will be ignored 2024-04-20T20:46:08.7722542Z pass 2024-04-20T20:46:16.3113017Z 2024-04-20T20:46:22.1146036Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:46:22.1146673Z loading model: 0it [00:05, ?it/s] 2024-04-20T20:46:22.1147215Z cuda train BartForCausalLM 2024-04-20T20:46:59.3642934Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:46:59.3650220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-04-20T20:46:59.3652529Z return super().forward(positions + self.offset) 2024-04-20T20:46:59.3655055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T20:46:59.3656911Z return F.embedding( 2024-04-20T20:46:59.3657386Z 2024-04-20T20:47:05.9639114Z pass 2024-04-20T20:47:11.2220619Z 2024-04-20T20:47:21.0864846Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:47:21.0866619Z loading model: 0it [00:09, ?it/s] 2024-04-20T20:47:21.0867636Z cuda train BartForConditionalGeneration 2024-04-20T20:48:30.9968006Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:48:30.9975691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-04-20T20:48:30.9977930Z return super().forward(positions + self.offset) 2024-04-20T20:48:30.9980189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T20:48:30.9982086Z return F.embedding( 2024-04-20T20:48:30.9982571Z 2024-04-20T20:48:40.2166281Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:48:40.2169821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-04-20T20:48:40.2171907Z return super().forward(positions + self.offset) 2024-04-20T20:48:40.2174246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T20:48:40.2175876Z return F.embedding( 2024-04-20T20:48:40.2176278Z 2024-04-20T20:48:59.1887763Z pass 2024-04-20T20:49:05.2739454Z 2024-04-20T20:49:08.3588004Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:49:08.3589965Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:49:08.3590567Z cuda train BertForMaskedLM 2024-04-20T20:49:48.2012099Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:49:48.2013988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T20:49:48.2015295Z pred = mod(**cloned_inputs) 2024-04-20T20:49:48.2021378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:49:48.2022478Z return forward_call(*args, **kwargs) 2024-04-20T20:49:48.2023786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-04-20T20:49:48.2024919Z outputs = self.bert( 2024-04-20T20:49:48.2026157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:49:48.2027291Z return forward_call(*args, **kwargs) 2024-04-20T20:49:48.2028611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-04-20T20:49:48.2029778Z embedding_output = self.embeddings( 2024-04-20T20:49:48.2030970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:49:48.2032048Z return forward_call(*args, **kwargs) 2024-04-20T20:49:48.2033744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-04-20T20:49:48.2034951Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:49:48.2035392Z 2024-04-20T20:49:48.4477433Z W0420 20:49:48.446000 140671182832832 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:50:26.9377313Z pass 2024-04-20T20:50:33.8839759Z 2024-04-20T20:50:36.6533763Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:50:36.6535368Z loading model: 0it [00:02, ?it/s] 2024-04-20T20:50:36.6540128Z cuda train BertForQuestionAnswering 2024-04-20T20:51:16.3947107Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:51:16.3948992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T20:51:16.3950289Z pred = mod(**cloned_inputs) 2024-04-20T20:51:16.3955199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:51:16.3956286Z return forward_call(*args, **kwargs) 2024-04-20T20:51:16.3957578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1846, in forward 2024-04-20T20:51:16.3958704Z outputs = self.bert( 2024-04-20T20:51:16.3959793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:51:16.3960839Z return forward_call(*args, **kwargs) 2024-04-20T20:51:16.3962172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-04-20T20:51:16.3963319Z embedding_output = self.embeddings( 2024-04-20T20:51:16.3964987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:51:16.3966031Z return forward_call(*args, **kwargs) 2024-04-20T20:51:16.3967323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-04-20T20:51:16.3968517Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:51:16.3968944Z 2024-04-20T20:51:16.5396761Z W0420 20:51:16.538000 140573306021056 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:51:54.6939190Z pass 2024-04-20T20:52:01.5943976Z 2024-04-20T20:52:41.5172050Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:52:41.5173530Z loading model: 0it [00:39, ?it/s] 2024-04-20T20:52:41.5174307Z cuda train BlenderbotForCausalLM 2024-04-20T20:52:41.5174969Z pass_due_to_skip 2024-04-20T20:52:46.3377802Z 2024-04-20T20:52:48.8407123Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:52:48.8407726Z loading model: 0it [00:02, ?it/s] 2024-04-20T20:52:48.8408300Z cuda train BlenderbotSmallForCausalLM 2024-04-20T20:52:48.8882659Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-04-20T20:53:19.6720997Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:53:19.6726186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-04-20T20:53:19.6727500Z return super().forward(positions) 2024-04-20T20:53:19.6728679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T20:53:19.6729685Z return F.embedding( 2024-04-20T20:53:19.6729946Z 2024-04-20T20:53:24.4257197Z pass 2024-04-20T20:53:29.7505337Z 2024-04-20T20:53:32.7174096Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:53:32.7175105Z loading model: 0it [00:02, ?it/s] 2024-04-20T20:53:32.7175725Z cuda train BlenderbotSmallForConditionalGeneration 2024-04-20T20:53:32.7769715Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-04-20T20:54:26.7995443Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:54:26.8000532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-04-20T20:54:26.8001869Z return super().forward(positions) 2024-04-20T20:54:26.8003040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T20:54:26.8004049Z return F.embedding( 2024-04-20T20:54:26.8004311Z 2024-04-20T20:54:33.1590638Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:54:33.1593849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-04-20T20:54:33.1595894Z return super().forward(positions) 2024-04-20T20:54:33.1597872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T20:54:33.1599420Z return F.embedding( 2024-04-20T20:54:33.1599691Z 2024-04-20T20:54:44.6099922Z pass 2024-04-20T20:54:50.0770551Z 2024-04-20T20:54:53.4375897Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:54:53.4376853Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:54:53.4377760Z cuda train CamemBert 2024-04-20T20:55:33.6647593Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:55:33.6649255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T20:55:33.6651580Z pred = mod(**cloned_inputs) 2024-04-20T20:55:33.6657907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:55:33.6659646Z return forward_call(*args, **kwargs) 2024-04-20T20:55:33.6661118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 979, in forward 2024-04-20T20:55:33.6662331Z outputs = self.roberta( 2024-04-20T20:55:33.6663428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:55:33.6664499Z return forward_call(*args, **kwargs) 2024-04-20T20:55:33.6665871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 881, in forward 2024-04-20T20:55:33.6667106Z embedding_output = self.embeddings( 2024-04-20T20:55:33.6668276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:55:33.6669334Z return forward_call(*args, **kwargs) 2024-04-20T20:55:33.6670701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 139, in forward 2024-04-20T20:55:33.6672024Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:55:33.6672451Z 2024-04-20T20:55:33.9172580Z W0420 20:55:33.915000 140186117612736 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:56:12.8612403Z pass 2024-04-20T20:56:20.0084078Z 2024-04-20T20:56:23.7797907Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:56:23.7798495Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:56:23.7799035Z cuda train DebertaForMaskedLM 2024-04-20T20:57:08.9172079Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:57:08.9173978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T20:57:08.9175287Z pred = mod(**cloned_inputs) 2024-04-20T20:57:08.9183305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:57:08.9184901Z return forward_call(*args, **kwargs) 2024-04-20T20:57:08.9186326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1062, in forward 2024-04-20T20:57:08.9187499Z outputs = self.deberta( 2024-04-20T20:57:08.9188629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:57:08.9189700Z return forward_call(*args, **kwargs) 2024-04-20T20:57:08.9191082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-04-20T20:57:08.9192338Z embedding_output = self.embeddings( 2024-04-20T20:57:08.9193521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:57:08.9194619Z return forward_call(*args, **kwargs) 2024-04-20T20:57:08.9195960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-04-20T20:57:08.9197203Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:57:08.9197633Z 2024-04-20T20:57:09.1691781Z W0420 20:57:09.167000 139934190265536 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:57:41.7435313Z pass 2024-04-20T20:57:48.7153980Z 2024-04-20T20:57:51.9166400Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:57:51.9169525Z loading model: 0it [00:03, ?it/s] 2024-04-20T20:57:51.9170169Z cuda train DebertaForQuestionAnswering 2024-04-20T20:58:37.2490104Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T20:58:37.2491632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T20:58:37.2494088Z pred = mod(**cloned_inputs) 2024-04-20T20:58:37.2498918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:58:37.2500019Z return forward_call(*args, **kwargs) 2024-04-20T20:58:37.2501383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1388, in forward 2024-04-20T20:58:37.2502576Z outputs = self.deberta( 2024-04-20T20:58:37.2503677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:58:37.2504742Z return forward_call(*args, **kwargs) 2024-04-20T20:58:37.2506153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-04-20T20:58:37.2507372Z embedding_output = self.embeddings( 2024-04-20T20:58:37.2508580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T20:58:37.2509620Z return forward_call(*args, **kwargs) 2024-04-20T20:58:37.2510952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-04-20T20:58:37.2512199Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T20:58:37.2512623Z 2024-04-20T20:58:37.3874772Z W0420 20:58:37.386000 139934387127488 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T20:59:08.9755727Z E0420 20:59:08.973000 139934387127488 torch/_dynamo/utils.py:1440] RMSE (res-fp64): 0.01709, (ref-fp64): 0.00506 and shape=torch.Size([2]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000 2024-04-20T20:59:08.9757720Z E0420 20:59:08.974000 139934387127488 torch/_dynamo/utils.py:1354] Accuracy failed for key name qa_outputs.bias 2024-04-20T20:59:08.9798500Z fail_accuracy 2024-04-20T20:59:15.9275476Z 2024-04-20T20:59:33.8118462Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:59:33.8119400Z loading model: 0it [00:17, ?it/s] 2024-04-20T20:59:33.8121515Z cuda train DebertaV2ForMaskedLM 2024-04-20T20:59:33.8122109Z pass_due_to_skip 2024-04-20T20:59:38.5224430Z 2024-04-20T20:59:53.4937326Z loading model: 0it [00:00, ?it/s] 2024-04-20T20:59:53.4938306Z loading model: 0it [00:14, ?it/s] 2024-04-20T20:59:53.4938876Z cuda train DebertaV2ForQuestionAnswering 2024-04-20T21:01:12.6399763Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:01:12.6401366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:01:12.6402860Z pred = mod(**cloned_inputs) 2024-04-20T21:01:12.6407705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:01:12.6408815Z return forward_call(*args, **kwargs) 2024-04-20T21:01:12.6410212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-04-20T21:01:12.6411500Z outputs = self.deberta( 2024-04-20T21:01:12.6412616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:01:12.6414051Z return forward_call(*args, **kwargs) 2024-04-20T21:01:12.6415463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1062, in forward 2024-04-20T21:01:12.6416696Z embedding_output = self.embeddings( 2024-04-20T21:01:12.6417892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:01:12.6418948Z return forward_call(*args, **kwargs) 2024-04-20T21:01:12.6430711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 874, in forward 2024-04-20T21:01:12.6432061Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:01:12.6432504Z 2024-04-20T21:01:14.4622919Z pass 2024-04-20T21:01:21.1582761Z 2024-04-20T21:01:23.5268154Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:01:23.5268723Z loading model: 0it [00:02, ?it/s] 2024-04-20T21:01:23.5269261Z cuda train DistilBertForMaskedLM 2024-04-20T21:01:47.4629434Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:01:47.4630964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:01:47.4632218Z pred = mod(**cloned_inputs) 2024-04-20T21:01:47.4640967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:01:47.4643013Z return forward_call(*args, **kwargs) 2024-04-20T21:01:47.4645584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-04-20T21:01:47.4647886Z dlbrt_output = self.distilbert( 2024-04-20T21:01:47.4649868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:01:47.4651726Z return forward_call(*args, **kwargs) 2024-04-20T21:01:47.4654392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-04-20T21:01:47.4657098Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-04-20T21:01:47.4659657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:01:47.4661389Z return forward_call(*args, **kwargs) 2024-04-20T21:01:47.4663281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-04-20T21:01:47.4664880Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-04-20T21:01:47.4665483Z 2024-04-20T21:01:47.6077011Z W0420 21:01:47.606000 140022326768832 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:02:08.0094059Z pass 2024-04-20T21:02:10.1463558Z accuracy pass_rate=81.25% 2024-04-20T21:02:10.1467477Z calls_captured gmean=0.00x mean=811.500x 2024-04-20T21:02:10.1468504Z unique_graphs gmean=0.00x mean=6.500x 2024-04-20T21:02:10.1469497Z graph_breaks gmean=0.00x mean=7.750x 2024-04-20T21:02:10.1470476Z unique_graph_breaks gmean=0.00x mean=4.812x 2024-04-20T21:02:10.1472196Z autograd_captures gmean=0.00x mean=0.000x 2024-04-20T21:02:10.1474308Z autograd_compiles gmean=0.00x mean=0.000x 2024-04-20T21:02:10.7612198Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *dynamic-true* ]] 2024-04-20T21:02:10.7616512Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --training --amp --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 3 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_huggingface_amp_training_cuda_accuracy.csv 2024-04-20T21:02:18.6756845Z 2024-04-20T21:02:22.9705684Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:02:22.9706721Z loading model: 0it [00:04, ?it/s] 2024-04-20T21:02:22.9707268Z cuda train AlbertForMaskedLM 2024-04-20T21:03:02.1648589Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:03:02.1650294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:03:02.1653491Z pred = mod(**cloned_inputs) 2024-04-20T21:03:02.1657621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:03:02.1658747Z return forward_call(*args, **kwargs) 2024-04-20T21:03:02.1660101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2024-04-20T21:03:02.1661238Z outputs = self.albert( 2024-04-20T21:03:02.1662340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:03:02.1663400Z return forward_call(*args, **kwargs) 2024-04-20T21:03:02.1664707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-04-20T21:03:02.1665869Z embedding_output = self.embeddings( 2024-04-20T21:03:02.1667090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:03:02.1668163Z return forward_call(*args, **kwargs) 2024-04-20T21:03:02.1669471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-04-20T21:03:02.1670681Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:03:02.1671116Z 2024-04-20T21:03:03.3830034Z pass 2024-04-20T21:03:09.4893930Z 2024-04-20T21:03:13.9082758Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:03:13.9083352Z loading model: 0it [00:04, ?it/s] 2024-04-20T21:03:13.9083891Z cuda train AlbertForQuestionAnswering 2024-04-20T21:03:53.5199374Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:03:53.5201251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:03:53.5203036Z pred = mod(**cloned_inputs) 2024-04-20T21:03:53.5207451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:03:53.5208544Z return forward_call(*args, **kwargs) 2024-04-20T21:03:53.5210365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 1256, in forward 2024-04-20T21:03:53.5211603Z outputs = self.albert( 2024-04-20T21:03:53.5212721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:03:53.5213965Z return forward_call(*args, **kwargs) 2024-04-20T21:03:53.5215265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-04-20T21:03:53.5216494Z embedding_output = self.embeddings( 2024-04-20T21:03:53.5217685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:03:53.5218742Z return forward_call(*args, **kwargs) 2024-04-20T21:03:53.5220044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-04-20T21:03:53.5221264Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:03:53.5221710Z 2024-04-20T21:03:53.8895798Z W0420 21:03:53.888000 140132508759232 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:03:59.7565325Z pass 2024-04-20T21:04:06.2835245Z 2024-04-20T21:04:10.5719203Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:04:10.5720110Z loading model: 0it [00:04, ?it/s] 2024-04-20T21:04:10.5721119Z cuda train AllenaiLongformerBase 2024-04-20T21:04:12.8109916Z W0420 21:04:12.809000 139669579334848 torch/_dynamo/variables/tensor.py:687] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-04-20T21:04:12.8111728Z W0420 21:04:12.809000 139669579334848 torch/_dynamo/variables/tensor.py:687] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-04-20T21:04:12.8113824Z W0420 21:04:12.809000 139669579334848 torch/_dynamo/variables/tensor.py:687] [3/0] or: 2024-04-20T21:04:12.8115179Z W0420 21:04:12.809000 139669579334848 torch/_dynamo/variables/tensor.py:687] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-04-20T21:04:12.8116941Z W0420 21:04:12.809000 139669579334848 torch/_dynamo/variables/tensor.py:687] [3/0] to include these operations in the captured graph. 2024-04-20T21:04:12.8118277Z W0420 21:04:12.809000 139669579334848 torch/_dynamo/variables/tensor.py:687] [3/0] 2024-04-20T21:05:53.8035088Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:05:53.8044606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in torch_dynamo_resume_in_forward_at_1291 2024-04-20T21:05:53.8046125Z layer_outputs = layer_module( 2024-04-20T21:05:53.8047336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:05:53.8048415Z return forward_call(*args, **kwargs) 2024-04-20T21:05:53.8049812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-04-20T21:05:53.8051063Z self_attn_outputs = self.attention( 2024-04-20T21:05:53.8076353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:05:53.8077507Z return forward_call(*args, **kwargs) 2024-04-20T21:05:53.8078902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-04-20T21:05:53.8080139Z self_outputs = self.self( 2024-04-20T21:05:53.8081272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:05:53.8082330Z return forward_call(*args, **kwargs) 2024-04-20T21:05:53.8083721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 571, in forward 2024-04-20T21:05:53.8085038Z attn_scores = self._sliding_chunks_query_key_matmul( 2024-04-20T21:05:53.8087172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 836, in _sliding_chunks_query_key_matmul 2024-04-20T21:05:53.8088882Z query = self._chunk(query, window_overlap, getattr(self.config, "onnx_export", False)) 2024-04-20T21:05:53.8090573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 778, in _chunk 2024-04-20T21:05:53.8091979Z return hidden_states.as_strided(size=chunk_size, stride=chunk_stride) 2024-04-20T21:05:53.8092556Z 2024-04-20T21:05:54.2546725Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:05:54.2548792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1734, in forward 2024-04-20T21:05:54.2550074Z embedding_output = self.embeddings( 2024-04-20T21:05:54.2551287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:05:54.2552374Z return forward_call(*args, **kwargs) 2024-04-20T21:05:54.2553771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 470, in forward 2024-04-20T21:05:54.2555067Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:05:54.2555496Z 2024-04-20T21:05:54.2649015Z W0420 21:05:54.263000 139669579334848 torch/_logging/_internal.py:1016] [16/0] Profiler function will be ignored 2024-04-20T21:06:37.3586063Z pass 2024-04-20T21:06:44.8555502Z 2024-04-20T21:06:50.7234122Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:06:50.7237017Z loading model: 0it [00:05, ?it/s] 2024-04-20T21:06:50.7238662Z cuda train BartForCausalLM 2024-04-20T21:07:28.4576121Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:07:28.4580610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-04-20T21:07:28.4581942Z return super().forward(positions + self.offset) 2024-04-20T21:07:28.4583160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T21:07:28.4584157Z return F.embedding( 2024-04-20T21:07:28.4584433Z 2024-04-20T21:07:35.4306389Z pass 2024-04-20T21:07:41.0783779Z 2024-04-20T21:07:51.0519886Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:07:51.0520477Z loading model: 0it [00:09, ?it/s] 2024-04-20T21:07:51.0521044Z cuda train BartForConditionalGeneration 2024-04-20T21:09:02.4069674Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:09:02.4074452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-04-20T21:09:02.4076473Z return super().forward(positions + self.offset) 2024-04-20T21:09:02.4078704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T21:09:02.4080173Z return F.embedding( 2024-04-20T21:09:02.4080455Z 2024-04-20T21:09:11.8209260Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:09:11.8211037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-04-20T21:09:11.8212311Z return super().forward(positions + self.offset) 2024-04-20T21:09:11.8213684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T21:09:11.8214679Z return F.embedding( 2024-04-20T21:09:11.8214976Z 2024-04-20T21:09:32.8075990Z pass 2024-04-20T21:09:38.9038382Z 2024-04-20T21:09:42.0411874Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:09:42.0412791Z loading model: 0it [00:03, ?it/s] 2024-04-20T21:09:42.0415588Z cuda train BertForMaskedLM 2024-04-20T21:10:23.2438288Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:10:23.2440919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:10:23.2443085Z pred = mod(**cloned_inputs) 2024-04-20T21:10:23.2450225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:10:23.2452091Z return forward_call(*args, **kwargs) 2024-04-20T21:10:23.2454628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-04-20T21:10:23.2456634Z outputs = self.bert( 2024-04-20T21:10:23.2458576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:10:23.2460398Z return forward_call(*args, **kwargs) 2024-04-20T21:10:23.2462666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-04-20T21:10:23.2464684Z embedding_output = self.embeddings( 2024-04-20T21:10:23.2466725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:10:23.2468460Z return forward_call(*args, **kwargs) 2024-04-20T21:10:23.2470153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-04-20T21:10:23.2471342Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:10:23.2471926Z 2024-04-20T21:10:23.4924623Z W0420 21:10:23.491000 139673847002304 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:11:02.7883530Z pass 2024-04-20T21:11:09.7989750Z 2024-04-20T21:11:12.5763177Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:11:12.5764099Z loading model: 0it [00:02, ?it/s] 2024-04-20T21:11:12.5765650Z cuda train BertForQuestionAnswering 2024-04-20T21:11:52.4659632Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:11:52.4661747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:11:52.4663972Z pred = mod(**cloned_inputs) 2024-04-20T21:11:52.4671668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:11:52.4673596Z return forward_call(*args, **kwargs) 2024-04-20T21:11:52.4675816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1846, in forward 2024-04-20T21:11:52.4678138Z outputs = self.bert( 2024-04-20T21:11:52.4680043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:11:52.4681899Z return forward_call(*args, **kwargs) 2024-04-20T21:11:52.4684176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-04-20T21:11:52.4686181Z embedding_output = self.embeddings( 2024-04-20T21:11:52.4688225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:11:52.4690063Z return forward_call(*args, **kwargs) 2024-04-20T21:11:52.4692235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-04-20T21:11:52.4694089Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:11:52.4694535Z 2024-04-20T21:11:52.6075818Z W0420 21:11:52.606000 139709356737728 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:12:30.8808608Z pass 2024-04-20T21:12:37.8281363Z 2024-04-20T21:13:17.7285127Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:13:17.7285745Z loading model: 0it [00:39, ?it/s] 2024-04-20T21:13:17.7286321Z cuda train BlenderbotForCausalLM 2024-04-20T21:13:17.7286881Z pass_due_to_skip 2024-04-20T21:13:22.4784227Z 2024-04-20T21:13:24.9225199Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:13:24.9225880Z loading model: 0it [00:02, ?it/s] 2024-04-20T21:13:24.9226524Z cuda train BlenderbotSmallForCausalLM 2024-04-20T21:13:24.9492635Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-04-20T21:13:55.9015537Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:13:55.9021594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-04-20T21:13:55.9022948Z return super().forward(positions) 2024-04-20T21:13:55.9024107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T21:13:55.9025108Z return F.embedding( 2024-04-20T21:13:55.9025549Z 2024-04-20T21:14:00.5216833Z pass 2024-04-20T21:14:05.8607413Z 2024-04-20T21:14:08.8379242Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:14:08.8379831Z loading model: 0it [00:02, ?it/s] 2024-04-20T21:14:08.8380438Z cuda train BlenderbotSmallForConditionalGeneration 2024-04-20T21:14:08.9213990Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-04-20T21:15:02.5072686Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:15:02.5080110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-04-20T21:15:02.5083635Z return super().forward(positions) 2024-04-20T21:15:02.5085756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T21:15:02.5087588Z return F.embedding( 2024-04-20T21:15:02.5088116Z 2024-04-20T21:15:08.8060763Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:15:08.8063802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-04-20T21:15:08.8066089Z return super().forward(positions) 2024-04-20T21:15:08.8067932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward 2024-04-20T21:15:08.8069499Z return F.embedding( 2024-04-20T21:15:08.8069928Z 2024-04-20T21:15:19.8684718Z pass 2024-04-20T21:15:25.3286605Z 2024-04-20T21:15:28.4821530Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:15:28.4822136Z loading model: 0it [00:03, ?it/s] 2024-04-20T21:15:28.4822647Z cuda train CamemBert 2024-04-20T21:16:08.2751896Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:16:08.2753669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:16:08.2755873Z pred = mod(**cloned_inputs) 2024-04-20T21:16:08.2760933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:16:08.2762095Z return forward_call(*args, **kwargs) 2024-04-20T21:16:08.2763487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 979, in forward 2024-04-20T21:16:08.2764697Z outputs = self.roberta( 2024-04-20T21:16:08.2765811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:16:08.2766878Z return forward_call(*args, **kwargs) 2024-04-20T21:16:08.2768240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 881, in forward 2024-04-20T21:16:08.2769916Z embedding_output = self.embeddings( 2024-04-20T21:16:08.2771119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:16:08.2772211Z return forward_call(*args, **kwargs) 2024-04-20T21:16:08.2773898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 139, in forward 2024-04-20T21:16:08.2775191Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:16:08.2775613Z 2024-04-20T21:16:08.5232022Z W0420 21:16:08.522000 139699833803968 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:16:47.0988972Z pass 2024-04-20T21:16:54.1922134Z 2024-04-20T21:16:57.8939185Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:16:57.8940101Z loading model: 0it [00:03, ?it/s] 2024-04-20T21:16:57.8940965Z cuda train DebertaForMaskedLM 2024-04-20T21:17:42.8797372Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:17:42.8799042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:17:42.8801269Z pred = mod(**cloned_inputs) 2024-04-20T21:17:42.8805220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:17:42.8806350Z return forward_call(*args, **kwargs) 2024-04-20T21:17:42.8807737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1062, in forward 2024-04-20T21:17:42.8808911Z outputs = self.deberta( 2024-04-20T21:17:42.8810510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:17:42.8811567Z return forward_call(*args, **kwargs) 2024-04-20T21:17:42.8813209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-04-20T21:17:42.8814424Z embedding_output = self.embeddings( 2024-04-20T21:17:42.8815633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:17:42.8816754Z return forward_call(*args, **kwargs) 2024-04-20T21:17:42.8818086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-04-20T21:17:42.8819320Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:17:42.8819760Z 2024-04-20T21:17:43.1257413Z W0420 21:17:43.124000 140120861463744 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:18:14.9130513Z pass 2024-04-20T21:18:21.8662362Z 2024-04-20T21:18:25.1482292Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:18:25.1483115Z loading model: 0it [00:03, ?it/s] 2024-04-20T21:18:25.1483876Z cuda train DebertaForQuestionAnswering 2024-04-20T21:19:09.6445207Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:19:09.6447162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:19:09.6448480Z pred = mod(**cloned_inputs) 2024-04-20T21:19:09.6453812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:19:09.6454900Z return forward_call(*args, **kwargs) 2024-04-20T21:19:09.6456256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1388, in forward 2024-04-20T21:19:09.6457486Z outputs = self.deberta( 2024-04-20T21:19:09.6458604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:19:09.6460129Z return forward_call(*args, **kwargs) 2024-04-20T21:19:09.6461532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-04-20T21:19:09.6462741Z embedding_output = self.embeddings( 2024-04-20T21:19:09.6463934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:19:09.6465006Z return forward_call(*args, **kwargs) 2024-04-20T21:19:09.6466337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-04-20T21:19:09.6467569Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:19:09.6468009Z 2024-04-20T21:19:09.7816662Z W0420 21:19:09.780000 139789104710848 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:19:41.1120893Z E0420 21:19:41.110000 139789104710848 torch/_dynamo/utils.py:1440] RMSE (res-fp64): 0.01709, (ref-fp64): 0.00506 and shape=torch.Size([2]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000 2024-04-20T21:19:41.1122871Z E0420 21:19:41.110000 139789104710848 torch/_dynamo/utils.py:1354] Accuracy failed for key name qa_outputs.bias 2024-04-20T21:19:41.1175331Z fail_accuracy 2024-04-20T21:19:47.8406605Z 2024-04-20T21:20:05.4853497Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:20:05.4854082Z loading model: 0it [00:17, ?it/s] 2024-04-20T21:20:05.4854609Z cuda train DebertaV2ForMaskedLM 2024-04-20T21:20:05.4856224Z pass_due_to_skip 2024-04-20T21:20:10.1037102Z 2024-04-20T21:20:25.1289868Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:20:25.1290636Z loading model: 0it [00:15, ?it/s] 2024-04-20T21:20:25.1292675Z cuda train DebertaV2ForQuestionAnswering 2024-04-20T21:21:43.6144032Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:21:43.6146757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:21:43.6148762Z pred = mod(**cloned_inputs) 2024-04-20T21:21:43.6155026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:21:43.6156185Z return forward_call(*args, **kwargs) 2024-04-20T21:21:43.6157600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-04-20T21:21:43.6158798Z outputs = self.deberta( 2024-04-20T21:21:43.6159911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:21:43.6160976Z return forward_call(*args, **kwargs) 2024-04-20T21:21:43.6162383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1062, in forward 2024-04-20T21:21:43.6163609Z embedding_output = self.embeddings( 2024-04-20T21:21:43.6164801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:21:43.6165861Z return forward_call(*args, **kwargs) 2024-04-20T21:21:43.6167258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 874, in forward 2024-04-20T21:21:43.6168519Z inputs_embeds = self.word_embeddings(input_ids) 2024-04-20T21:21:43.6168958Z 2024-04-20T21:21:45.3716839Z pass 2024-04-20T21:21:51.9735422Z 2024-04-20T21:21:54.4277835Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:21:54.4278432Z loading model: 0it [00:02, ?it/s] 2024-04-20T21:21:54.4278987Z cuda train DistilBertForMaskedLM 2024-04-20T21:22:18.1989147Z skipping cudagraphs due to deterministic index put. Found from : 2024-04-20T21:22:18.1992309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 574, in torch_dynamo_resume_in_forward_and_backward_pass_at_572 2024-04-20T21:22:18.1994280Z pred = mod(**cloned_inputs) 2024-04-20T21:22:18.2001782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:22:18.2003893Z return forward_call(*args, **kwargs) 2024-04-20T21:22:18.2006528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-04-20T21:22:18.2008956Z dlbrt_output = self.distilbert( 2024-04-20T21:22:18.2011280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:22:18.2013587Z return forward_call(*args, **kwargs) 2024-04-20T21:22:18.2016237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-04-20T21:22:18.2019119Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-04-20T21:22:18.2021976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T21:22:18.2023912Z return forward_call(*args, **kwargs) 2024-04-20T21:22:18.2026474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-04-20T21:22:18.2029004Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-04-20T21:22:18.2030066Z 2024-04-20T21:22:18.3501388Z W0420 21:22:18.348000 140477599503552 torch/_logging/_internal.py:1016] [4/0] Profiler function will be ignored 2024-04-20T21:22:38.6861123Z pass 2024-04-20T21:22:40.8515209Z accuracy pass_rate=81.25% 2024-04-20T21:22:40.8516401Z calls_captured gmean=0.00x mean=811.500x 2024-04-20T21:22:40.8519074Z unique_graphs gmean=0.00x mean=6.500x 2024-04-20T21:22:40.8522002Z graph_breaks gmean=0.00x mean=7.750x 2024-04-20T21:22:40.8525070Z unique_graph_breaks gmean=0.00x mean=4.812x 2024-04-20T21:22:40.8527142Z autograd_captures gmean=0.00x mean=0.000x 2024-04-20T21:22:40.8529814Z autograd_compiles gmean=0.00x mean=0.000x 2024-04-20T21:22:41.4586191Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *cppwrapper-true* ]] 2024-04-20T21:22:41.4590073Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *freezing_cudagraphs-true* ]] 2024-04-20T21:22:41.4592869Z + [[ training == \i\n\f\e\r\e\n\c\e ]] 2024-04-20T21:22:41.4594958Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *freeze_autotune_cudagraphs-true* ]] 2024-04-20T21:22:41.4598269Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *aotinductor-true* ]] 2024-04-20T21:22:41.4600032Z + [[ training == \i\n\f\e\r\e\n\c\e ]] 2024-04-20T21:22:41.4602007Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *maxautotune-true* ]] 2024-04-20T21:22:41.4605271Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *cudagraphs_low_precision-true* ]] 2024-04-20T21:22:41.4607122Z + [[ training == \i\n\f\e\r\e\n\c\e ]] 2024-04-20T21:22:41.4607640Z + for target in "${targets[@]}" 2024-04-20T21:22:41.4608159Z + target_flag=("--${target}") 2024-04-20T21:22:41.4608599Z + local target_flag 2024-04-20T21:22:41.4609021Z + [[ performance == \p\e\r\f\o\r\m\a\n\c\e ]] 2024-04-20T21:22:41.4609651Z + target_flag+=(--cold-start-latency) 2024-04-20T21:22:41.4611990Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *default-true* ]] 2024-04-20T21:22:41.4616286Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_amp_training_cuda_performance.csv 2024-04-20T21:22:49.2363420Z 2024-04-20T21:22:53.9494298Z loading model: 0it [00:00, ?it/s] 2024-04-20T21:22:53.9494877Z loading model: 0it [00:04, ?it/s] 2024-04-20T21:22:53.9495420Z cuda train AlbertForMaskedLM 2024-04-20T21:23:51.0301321Z 2024-04-20T21:23:51.6923641Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:25:21.8523529Z 2024-04-20T21:25:22.5204367Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:28:38.8372144Z 2024-04-20T21:28:39.1387066Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:33:31.9998353Z 2024-04-20T21:33:32.2372803Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:35:19.1861462Z 2024-04-20T21:35:19.3482139Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:40:30.2684183Z 2024-04-20T21:40:30.5040525Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:42:23.7935754Z 2024-04-20T21:42:23.9680815Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:44:14.0567498Z 2024-04-20T21:44:14.3277261Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:47:43.3530995Z 2024-04-20T21:47:43.6424907Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:51:11.2124126Z 2024-04-20T21:51:11.4724142Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:54:03.2360319Z 2024-04-20T21:54:04.1083473Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T21:57:30.7032161Z 2024-04-20T21:57:31.0511897Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:02:53.7971068Z 2024-04-20T22:02:54.0924220Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:04:44.7298273Z 2024-04-20T22:04:44.9695284Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:10:17.0753454Z 2024-04-20T22:10:17.3863994Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:12:12.9407772Z 2024-04-20T22:12:13.1909929Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:14:05.8499270Z 2024-04-20T22:14:06.1870237Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:17:41.4245003Z 2024-04-20T22:17:41.9043860Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:21:12.4694095Z 2024-04-20T22:21:12.7846625Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:24:54.0931668Z 2024-04-20T22:24:54.9783653Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:28:57.4282026Z 2024-04-20T22:28:57.7798393Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:33:37.5507552Z 2024-04-20T22:33:37.8959030Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:35:59.7720996Z 2024-04-20T22:36:00.0578234Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:40:54.9074054Z 2024-04-20T22:40:55.2886849Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:43:19.5460283Z 2024-04-20T22:43:19.8011260Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:45:38.9753285Z 2024-04-20T22:45:39.3136453Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:50:11.8930248Z 2024-04-20T22:50:12.3769856Z running benchmark: 0% 0/30 [00:00 will be ignored 2024-04-20T22:53:57.2958477Z 2024-04-20T22:53:57.6090509Z running benchmark: 0% 0/30 [00:00at(28); 2024-04-20T23:33:05.8240077Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:33:16.8001974Z pass 2024-04-20T23:33:22.3208919Z 2024-04-20T23:33:26.4653118Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:33:26.4656462Z loading model: 0it [00:04, ?it/s] 2024-04-20T23:33:26.4663215Z cuda eval AlbertForQuestionAnswering 2024-04-20T23:34:01.9754024Z /tmp/torchinductor_jenkins/cvzzjfshfn4yol22x6yyjdhkdmrhb2np5pt2ekifc4l65234ckza/ccdifapil45mun2hykhrp5oraqyhi22mnlvkwlwpl2gaa6n74345.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-04-20T23:34:01.9762631Z /tmp/torchinductor_jenkins/cvzzjfshfn4yol22x6yyjdhkdmrhb2np5pt2ekifc4l65234ckza/ccdifapil45mun2hykhrp5oraqyhi22mnlvkwlwpl2gaa6n74345.cpp:830:10: warning: variable ‘albert_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:34:01.9765498Z 830 | auto albert_embeddings_token_type_ids = constants_->at(24); 2024-04-20T23:34:01.9766289Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:34:12.8786145Z pass 2024-04-20T23:34:18.7788090Z 2024-04-20T23:34:22.9466966Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:34:22.9467559Z loading model: 0it [00:04, ?it/s] 2024-04-20T23:34:22.9542171Z cuda eval AllenaiLongformerBase 2024-04-20T23:34:24.3293197Z ERROR:common:Consider annotating your code using torch._constrain_as_*(). Could not guard on data-dependent expression Eq(u0, 1) (unhinted: Eq(u0, 1)). (Size-like symbols: none) 2024-04-20T23:34:24.3297318Z 2024-04-20T23:34:24.3297938Z Potential framework code culprit (scroll up for full backtrace): 2024-04-20T23:34:24.3300592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 992, in evaluate_expr 2024-04-20T23:34:24.3302211Z return guard_scalar(self.sym_num) 2024-04-20T23:34:24.3302566Z 2024-04-20T23:34:24.3302824Z For more information, run with TORCH_LOGS="dynamic" 2024-04-20T23:34:24.3303799Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-04-20T23:34:24.3305023Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-04-20T23:34:24.3307601Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-04-20T23:34:24.3308635Z 2024-04-20T23:34:24.3308836Z User Stack (most recent call last): 2024-04-20T23:34:24.3309353Z (snipped, see stack below for prefix) 2024-04-20T23:34:24.3310826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-04-20T23:34:24.3312069Z outputs = self.longformer( 2024-04-20T23:34:24.3313209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3314272Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3315672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-04-20T23:34:24.3316963Z encoder_outputs = self.encoder( 2024-04-20T23:34:24.3318128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3319195Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3320681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-04-20T23:34:24.3322482Z layer_outputs = layer_module( 2024-04-20T23:34:24.3323667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3324733Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3326107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-04-20T23:34:24.3327417Z self_attn_outputs = self.attention( 2024-04-20T23:34:24.3328602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3329690Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3331346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-04-20T23:34:24.3332578Z self_outputs = self.self( 2024-04-20T23:34:24.3333834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3334940Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3336314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-04-20T23:34:24.3337560Z if is_global_attn: 2024-04-20T23:34:24.3337826Z 2024-04-20T23:34:24.3338131Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-04-20T23:34:24.3339614Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-04-20T23:34:24.3340636Z 2024-04-20T23:34:24.3340786Z from user code: 2024-04-20T23:34:24.3342037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-04-20T23:34:24.3343281Z outputs = self.longformer( 2024-04-20T23:34:24.3344415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3345483Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3346891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-04-20T23:34:24.3348129Z encoder_outputs = self.encoder( 2024-04-20T23:34:24.3349290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3350364Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3351900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-04-20T23:34:24.3353129Z layer_outputs = layer_module( 2024-04-20T23:34:24.3354274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3355341Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3356771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-04-20T23:34:24.3358014Z self_attn_outputs = self.attention( 2024-04-20T23:34:24.3359199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3360272Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3361667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-04-20T23:34:24.3362890Z self_outputs = self.self( 2024-04-20T23:34:24.3364004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3365079Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3366485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-04-20T23:34:24.3367682Z if is_global_attn: 2024-04-20T23:34:24.3367934Z 2024-04-20T23:34:24.3368290Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-04-20T23:34:24.3369021Z Traceback (most recent call last): 2024-04-20T23:34:24.3370243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 992, in evaluate_expr 2024-04-20T23:34:24.3371356Z return guard_scalar(self.sym_num) 2024-04-20T23:34:24.3372635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 505, in guard_scalar 2024-04-20T23:34:24.3373893Z return guard_bool(a) 2024-04-20T23:34:24.3375239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 732, in guard_bool 2024-04-20T23:34:24.3376615Z return a.node.guard_bool("", 0) # NB: uses Python backtrace 2024-04-20T23:34:24.3377991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 377, in guard_bool 2024-04-20T23:34:24.3379330Z r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) 2024-04-20T23:34:24.3380802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/recording.py", line 235, in wrapper 2024-04-20T23:34:24.3381882Z return fn(*args, **kwargs) 2024-04-20T23:34:24.3383148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4308, in evaluate_expr 2024-04-20T23:34:24.3384340Z raise self._make_data_dependent_error( 2024-04-20T23:34:24.3386014Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 1) (unhinted: Eq(u0, 1)). (Size-like symbols: none) 2024-04-20T23:34:24.3387330Z 2024-04-20T23:34:24.3387668Z Potential framework code culprit (scroll up for full backtrace): 2024-04-20T23:34:24.3389088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 992, in evaluate_expr 2024-04-20T23:34:24.3390180Z return guard_scalar(self.sym_num) 2024-04-20T23:34:24.3390541Z 2024-04-20T23:34:24.3390775Z For more information, run with TORCH_LOGS="dynamic" 2024-04-20T23:34:24.3391751Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-04-20T23:34:24.3392956Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-04-20T23:34:24.3394462Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-04-20T23:34:24.3395615Z 2024-04-20T23:34:24.3395783Z User Stack (most recent call last): 2024-04-20T23:34:24.3396318Z (snipped, see stack below for prefix) 2024-04-20T23:34:24.3397764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-04-20T23:34:24.3399013Z outputs = self.longformer( 2024-04-20T23:34:24.3400127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3401185Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3402573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-04-20T23:34:24.3403808Z encoder_outputs = self.encoder( 2024-04-20T23:34:24.3404959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3406023Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3407436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-04-20T23:34:24.3408669Z layer_outputs = layer_module( 2024-04-20T23:34:24.3409807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3410847Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3412224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-04-20T23:34:24.3413607Z self_attn_outputs = self.attention( 2024-04-20T23:34:24.3414789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3415834Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3417250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-04-20T23:34:24.3418475Z self_outputs = self.self( 2024-04-20T23:34:24.3419719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3420771Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3422156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-04-20T23:34:24.3423364Z if is_global_attn: 2024-04-20T23:34:24.3423615Z 2024-04-20T23:34:24.3423936Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-04-20T23:34:24.3424439Z 2024-04-20T23:34:24.3424808Z During handling of the above exception, another exception occurred: 2024-04-20T23:34:24.3425360Z 2024-04-20T23:34:24.3425527Z Traceback (most recent call last): 2024-04-20T23:34:24.3426600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2535, in check_accuracy 2024-04-20T23:34:24.3427554Z optimized_model_iter_fn = optimize_ctx( 2024-04-20T23:34:24.3428546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in export_aot_inductor 2024-04-20T23:34:24.3429707Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-04-20T23:34:24.3430766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1155, in load 2024-04-20T23:34:24.3431619Z gm = torch.export._trace._export( 2024-04-20T23:34:24.3432723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 833, in wrapper 2024-04-20T23:34:24.3433647Z raise e 2024-04-20T23:34:24.3434572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 816, in wrapper 2024-04-20T23:34:24.3435545Z ep = fn(*args, **kwargs) 2024-04-20T23:34:24.3436686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/exported_program.py", line 85, in wrapper 2024-04-20T23:34:24.3437864Z return fn(*args, **kwargs) 2024-04-20T23:34:24.3438913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 1075, in _export 2024-04-20T23:34:24.3439926Z gm_torch_level = _export_to_torch_ir( 2024-04-20T23:34:24.3441141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 428, in _export_to_torch_ir 2024-04-20T23:34:24.3442229Z gm_torch_level, _ = torch._dynamo.export( 2024-04-20T23:34:24.3443377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1230, in inner 2024-04-20T23:34:24.3444403Z result_traced = opt_f(*args, **kwargs) 2024-04-20T23:34:24.3445673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-04-20T23:34:24.3446824Z return self._call_impl(*args, **kwargs) 2024-04-20T23:34:24.3448025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3449092Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3450197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 403, in _fn 2024-04-20T23:34:24.3451168Z return fn(*args, **kwargs) 2024-04-20T23:34:24.3452343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-04-20T23:34:24.3453600Z return self._call_impl(*args, **kwargs) 2024-04-20T23:34:24.3454802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3455861Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3457092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 977, in catch_errors 2024-04-20T23:34:24.3458322Z return callback(frame, cache_entry, hooks, frame_state, skip=1) 2024-04-20T23:34:24.3459772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 411, in _convert_frame_assert 2024-04-20T23:34:24.3460890Z return _compile( 2024-04-20T23:34:24.3462109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 70, in wrapper_function 2024-04-20T23:34:24.3463171Z return function(*args, **kwargs) 2024-04-20T23:34:24.3463946Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-04-20T23:34:24.3464733Z return func(*args, **kwds) 2024-04-20T23:34:24.3465831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 700, in _compile 2024-04-20T23:34:24.3467065Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-04-20T23:34:24.3468363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 267, in time_wrapper 2024-04-20T23:34:24.3470181Z r = func(*args, **kwargs) 2024-04-20T23:34:24.3471342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 568, in compile_inner 2024-04-20T23:34:24.3472507Z out_code = transform_code_object(code, transform) 2024-04-20T23:34:24.3473958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1116, in transform_code_object 2024-04-20T23:34:24.3475222Z transformations(instructions, code_options) 2024-04-20T23:34:24.3476483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 173, in _fn 2024-04-20T23:34:24.3477466Z return fn(*args, **kwargs) 2024-04-20T23:34:24.3478593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 515, in transform 2024-04-20T23:34:24.3479613Z tracer.run() 2024-04-20T23:34:24.3480635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2237, in run 2024-04-20T23:34:24.3481822Z super().run() 2024-04-20T23:34:24.3482847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3483856Z while self.step(): 2024-04-20T23:34:24.3484906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3486003Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3487275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3488320Z return inner_fn(self, inst) 2024-04-20T23:34:24.3489546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1313, in CALL_FUNCTION_KW 2024-04-20T23:34:24.3490670Z self.call_function(fn, args, kwargs) 2024-04-20T23:34:24.3491904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3493217Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3494558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 352, in call_function 2024-04-20T23:34:24.3495687Z return tx.inline_user_function_return( 2024-04-20T23:34:24.3497076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3498523Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3500007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3501182Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3502493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3503566Z tracer.run() 2024-04-20T23:34:24.3504592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3505598Z while self.step(): 2024-04-20T23:34:24.3506814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3507918Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3509164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3510212Z return inner_fn(self, inst) 2024-04-20T23:34:24.3511420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1301, in CALL_FUNCTION_EX 2024-04-20T23:34:24.3512636Z self.call_function(fn, argsvars.items, kwargsvars) 2024-04-20T23:34:24.3513962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3515129Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3516477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 339, in call_function 2024-04-20T23:34:24.3517650Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3518969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function 2024-04-20T23:34:24.3520134Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3521443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function 2024-04-20T23:34:24.3522792Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-04-20T23:34:24.3524393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3525958Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3527490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3528646Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3529959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3531028Z tracer.run() 2024-04-20T23:34:24.3532048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3533174Z while self.step(): 2024-04-20T23:34:24.3534218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3535305Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3536576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3537618Z return inner_fn(self, inst) 2024-04-20T23:34:24.3538822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1313, in CALL_FUNCTION_KW 2024-04-20T23:34:24.3539964Z self.call_function(fn, args, kwargs) 2024-04-20T23:34:24.3541200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3542352Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3543663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 352, in call_function 2024-04-20T23:34:24.3544808Z return tx.inline_user_function_return( 2024-04-20T23:34:24.3546168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3547647Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3549115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3550419Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3551757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3552829Z tracer.run() 2024-04-20T23:34:24.3553845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3554838Z while self.step(): 2024-04-20T23:34:24.3555888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3557013Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3558255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3559295Z return inner_fn(self, inst) 2024-04-20T23:34:24.3560524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1301, in CALL_FUNCTION_EX 2024-04-20T23:34:24.3561735Z self.call_function(fn, argsvars.items, kwargsvars) 2024-04-20T23:34:24.3563048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3564187Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3565514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 339, in call_function 2024-04-20T23:34:24.3566725Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3568045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function 2024-04-20T23:34:24.3569204Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3570638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function 2024-04-20T23:34:24.3572004Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-04-20T23:34:24.3573731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3575183Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3576720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3577897Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3579230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3580297Z tracer.run() 2024-04-20T23:34:24.3581306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3582315Z while self.step(): 2024-04-20T23:34:24.3609444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3610569Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3611831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3613039Z return inner_fn(self, inst) 2024-04-20T23:34:24.3614280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1313, in CALL_FUNCTION_KW 2024-04-20T23:34:24.3615440Z self.call_function(fn, args, kwargs) 2024-04-20T23:34:24.3616716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3617895Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3619227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 352, in call_function 2024-04-20T23:34:24.3620667Z return tx.inline_user_function_return( 2024-04-20T23:34:24.3622028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3623473Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3624959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3626268Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3627616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3628688Z tracer.run() 2024-04-20T23:34:24.3629713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3630719Z while self.step(): 2024-04-20T23:34:24.3631762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3632859Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3634100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3635144Z return inner_fn(self, inst) 2024-04-20T23:34:24.3636372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1301, in CALL_FUNCTION_EX 2024-04-20T23:34:24.3637609Z self.call_function(fn, argsvars.items, kwargsvars) 2024-04-20T23:34:24.3638936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3640289Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3641620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 339, in call_function 2024-04-20T23:34:24.3642781Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3644105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function 2024-04-20T23:34:24.3645271Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3646612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function 2024-04-20T23:34:24.3647968Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-04-20T23:34:24.3649573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3651006Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3652493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3653820Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3655139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3656214Z tracer.run() 2024-04-20T23:34:24.3657279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3658297Z while self.step(): 2024-04-20T23:34:24.3659335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3660425Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3661664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3662709Z return inner_fn(self, inst) 2024-04-20T23:34:24.3663917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1313, in CALL_FUNCTION_KW 2024-04-20T23:34:24.3665207Z self.call_function(fn, args, kwargs) 2024-04-20T23:34:24.3666495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3667655Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3668985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 352, in call_function 2024-04-20T23:34:24.3670111Z return tx.inline_user_function_return( 2024-04-20T23:34:24.3671474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3672913Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3674402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3675562Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3676947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3678005Z tracer.run() 2024-04-20T23:34:24.3679017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3680010Z while self.step(): 2024-04-20T23:34:24.3681069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3682155Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3683381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3684538Z return inner_fn(self, inst) 2024-04-20T23:34:24.3685802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1301, in CALL_FUNCTION_EX 2024-04-20T23:34:24.3687060Z self.call_function(fn, argsvars.items, kwargsvars) 2024-04-20T23:34:24.3688380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3689535Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3690851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 339, in call_function 2024-04-20T23:34:24.3692035Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3693452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function 2024-04-20T23:34:24.3694625Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3695927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function 2024-04-20T23:34:24.3697320Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-04-20T23:34:24.3698931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3700371Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3701857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3703021Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3704336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3705403Z tracer.run() 2024-04-20T23:34:24.3706443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3707448Z while self.step(): 2024-04-20T23:34:24.3708641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3709742Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3710985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3712007Z return inner_fn(self, inst) 2024-04-20T23:34:24.3713233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1313, in CALL_FUNCTION_KW 2024-04-20T23:34:24.3714371Z self.call_function(fn, args, kwargs) 2024-04-20T23:34:24.3715612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3716789Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3718123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 352, in call_function 2024-04-20T23:34:24.3719263Z return tx.inline_user_function_return( 2024-04-20T23:34:24.3720627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3722070Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3723549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3724730Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3726048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3727191Z tracer.run() 2024-04-20T23:34:24.3728190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3729326Z while self.step(): 2024-04-20T23:34:24.3730380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3731475Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3732687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 492, in wrapper 2024-04-20T23:34:24.3733861Z return inner_fn(self, inst) 2024-04-20T23:34:24.3735082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1301, in CALL_FUNCTION_EX 2024-04-20T23:34:24.3736298Z self.call_function(fn, argsvars.items, kwargsvars) 2024-04-20T23:34:24.3737639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 730, in call_function 2024-04-20T23:34:24.3738800Z self.push(fn.call_function(self, args, kwargs)) 2024-04-20T23:34:24.3740133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 339, in call_function 2024-04-20T23:34:24.3741314Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3742641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 293, in call_function 2024-04-20T23:34:24.3743799Z return super().call_function(tx, args, kwargs) 2024-04-20T23:34:24.3745107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function 2024-04-20T23:34:24.3746579Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-04-20T23:34:24.3748174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 736, in inline_user_function_return 2024-04-20T23:34:24.3749602Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-04-20T23:34:24.3751098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2418, in inline_call 2024-04-20T23:34:24.3752274Z return cls.inline_call_(parent, func, args, kwargs) 2024-04-20T23:34:24.3753738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2534, in inline_call_ 2024-04-20T23:34:24.3754805Z tracer.run() 2024-04-20T23:34:24.3755826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run 2024-04-20T23:34:24.3756872Z while self.step(): 2024-04-20T23:34:24.3757933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step 2024-04-20T23:34:24.3759026Z self.dispatch_table[inst.opcode](self, inst) 2024-04-20T23:34:24.3760234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 446, in inner 2024-04-20T23:34:24.3761334Z eval_result = value.evaluate_expr(self.output) 2024-04-20T23:34:24.3762641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 994, in evaluate_expr 2024-04-20T23:34:24.3763754Z raise UserError( # noqa: TRY200 2024-04-20T23:34:24.3765493Z torch._dynamo.exc.UserError: Consider annotating your code using torch._constrain_as_*(). Could not guard on data-dependent expression Eq(u0, 1) (unhinted: Eq(u0, 1)). (Size-like symbols: none) 2024-04-20T23:34:24.3766880Z 2024-04-20T23:34:24.3767211Z Potential framework code culprit (scroll up for full backtrace): 2024-04-20T23:34:24.3768629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 992, in evaluate_expr 2024-04-20T23:34:24.3769736Z return guard_scalar(self.sym_num) 2024-04-20T23:34:24.3770083Z 2024-04-20T23:34:24.3770332Z For more information, run with TORCH_LOGS="dynamic" 2024-04-20T23:34:24.3771293Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-04-20T23:34:24.3772628Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-04-20T23:34:24.3774308Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-04-20T23:34:24.3775332Z 2024-04-20T23:34:24.3775517Z User Stack (most recent call last): 2024-04-20T23:34:24.3776057Z (snipped, see stack below for prefix) 2024-04-20T23:34:24.3777503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-04-20T23:34:24.3778742Z outputs = self.longformer( 2024-04-20T23:34:24.3779872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3780923Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3782321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-04-20T23:34:24.3783566Z encoder_outputs = self.encoder( 2024-04-20T23:34:24.3784724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3785829Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3787238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-04-20T23:34:24.3788461Z layer_outputs = layer_module( 2024-04-20T23:34:24.3789599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3790662Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3792043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-04-20T23:34:24.3793291Z self_attn_outputs = self.attention( 2024-04-20T23:34:24.3794479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3795535Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3797076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-04-20T23:34:24.3798315Z self_outputs = self.self( 2024-04-20T23:34:24.3799441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3800497Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3801866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-04-20T23:34:24.3803070Z if is_global_attn: 2024-04-20T23:34:24.3803338Z 2024-04-20T23:34:24.3803645Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-04-20T23:34:24.3805113Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-04-20T23:34:24.3806130Z 2024-04-20T23:34:24.3806281Z from user code: 2024-04-20T23:34:24.3807568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-04-20T23:34:24.3808805Z outputs = self.longformer( 2024-04-20T23:34:24.3809939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3811007Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3812375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-04-20T23:34:24.3813737Z encoder_outputs = self.encoder( 2024-04-20T23:34:24.3814898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3816145Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3817585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-04-20T23:34:24.3818828Z layer_outputs = layer_module( 2024-04-20T23:34:24.3819973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3821032Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3822405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-04-20T23:34:24.3823647Z self_attn_outputs = self.attention( 2024-04-20T23:34:24.3824832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3826031Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3827460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-04-20T23:34:24.3828675Z self_outputs = self.self( 2024-04-20T23:34:24.3829794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-04-20T23:34:24.3830858Z return forward_call(*args, **kwargs) 2024-04-20T23:34:24.3832239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-04-20T23:34:24.3833422Z if is_global_attn: 2024-04-20T23:34:24.3833690Z 2024-04-20T23:34:24.3834050Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-04-20T23:34:24.3834620Z 2024-04-20T23:34:24.3834976Z TorchDynamo optimized model failed to run because of following error 2024-04-20T23:34:24.3835707Z fail_to_run 2024-04-20T23:34:29.0539636Z 2024-04-20T23:34:34.7311576Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:34:34.7312151Z loading model: 0it [00:05, ?it/s] 2024-04-20T23:34:34.7394988Z cuda eval BartForCausalLM 2024-04-20T23:35:22.4143050Z pass 2024-04-20T23:35:28.1241567Z 2024-04-20T23:35:37.9862883Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:35:37.9865094Z loading model: 0it [00:09, ?it/s] 2024-04-20T23:35:38.0001984Z cuda eval BartForConditionalGeneration 2024-04-20T23:37:14.3148833Z pass 2024-04-20T23:37:20.6215335Z 2024-04-20T23:37:23.6845537Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:37:23.6846494Z loading model: 0it [00:03, ?it/s] 2024-04-20T23:37:23.6909184Z cuda eval BertForMaskedLM 2024-04-20T23:38:02.2599065Z /tmp/torchinductor_jenkins/cklxtfeugn2vzkatkoc67uacoluupuam45zfhwu65bazzrm6bfn6/c5qomyssscibcjrvg754l7krvdohe3amavwunjrk7gtfpuvanomn.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-04-20T23:38:02.2606761Z /tmp/torchinductor_jenkins/cklxtfeugn2vzkatkoc67uacoluupuam45zfhwu65bazzrm6bfn6/c5qomyssscibcjrvg754l7krvdohe3amavwunjrk7gtfpuvanomn.cpp:2428:10: warning: variable ‘bert_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:38:02.2611824Z 2428 | auto bert_embeddings_token_type_ids = constants_->at(202); 2024-04-20T23:38:02.2612649Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:38:08.3155090Z pass 2024-04-20T23:38:14.0324352Z 2024-04-20T23:38:16.8657274Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:38:16.8657846Z loading model: 0it [00:02, ?it/s] 2024-04-20T23:38:16.8711930Z cuda eval BertForQuestionAnswering 2024-04-20T23:38:55.2262198Z /tmp/torchinductor_jenkins/ckq2mbcorvt7r2ysjzx33lmk3kvhqb4ou27nofad7kwm33cbyqyp/c7zugfe5kwzjcuxv4pd6w2ugyk6eswg3q74szupta6wzrrya4hlf.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-04-20T23:38:55.2268648Z /tmp/torchinductor_jenkins/ckq2mbcorvt7r2ysjzx33lmk3kvhqb4ou27nofad7kwm33cbyqyp/c7zugfe5kwzjcuxv4pd6w2ugyk6eswg3q74szupta6wzrrya4hlf.cpp:2395:10: warning: variable ‘bert_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:38:55.2270978Z 2395 | auto bert_embeddings_token_type_ids = constants_->at(198); 2024-04-20T23:38:55.2271710Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:39:01.1159721Z pass 2024-04-20T23:39:06.5996957Z 2024-04-20T23:39:46.2558475Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:39:46.2559031Z loading model: 0it [00:39, ?it/s] 2024-04-20T23:39:46.2724945Z cuda eval BlenderbotForCausalLM 2024-04-20T23:39:46.2732539Z pass_due_to_skip 2024-04-20T23:39:51.0299770Z 2024-04-20T23:39:53.2552890Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:39:53.2553457Z loading model: 0it [00:02, ?it/s] 2024-04-20T23:39:53.2612069Z cuda eval BlenderbotSmallForCausalLM 2024-04-20T23:39:53.2862944Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-04-20T23:40:29.7708810Z pass 2024-04-20T23:40:35.5376771Z 2024-04-20T23:40:38.3941065Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:40:38.3942094Z loading model: 0it [00:02, ?it/s] 2024-04-20T23:40:38.4037937Z cuda eval BlenderbotSmallForConditionalGeneration 2024-04-20T23:40:38.4620173Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-04-20T23:41:44.6880861Z pass 2024-04-20T23:41:51.3472702Z 2024-04-20T23:41:54.5078773Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:41:54.5079729Z loading model: 0it [00:03, ?it/s] 2024-04-20T23:41:54.5140883Z cuda eval CamemBert 2024-04-20T23:42:35.4531421Z /tmp/torchinductor_jenkins/cpoyommzcuvrugmjqfihptlidgic473oievhdaos5t3p7upgccwi/c44dgqlumj54bngvkjc2knhphqeyljehxk5fm62uiipe2nkpcalf.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-04-20T23:42:35.4542207Z /tmp/torchinductor_jenkins/cpoyommzcuvrugmjqfihptlidgic473oievhdaos5t3p7upgccwi/c44dgqlumj54bngvkjc2knhphqeyljehxk5fm62uiipe2nkpcalf.cpp:2421:10: warning: variable ‘roberta_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:42:35.4544729Z 2421 | auto roberta_embeddings_token_type_ids = constants_->at(202); 2024-04-20T23:42:35.4545490Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:42:41.5942119Z pass 2024-04-20T23:42:48.1139761Z 2024-04-20T23:42:51.7574270Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:42:51.7575195Z loading model: 0it [00:03, ?it/s] 2024-04-20T23:42:51.7625134Z cuda eval DebertaForMaskedLM 2024-04-20T23:43:33.3101650Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-04-20T23:43:33.3106471Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1927:10: warning: variable ‘deberta_encoder_layer_0_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3110077Z 1927 | auto deberta_encoder_layer_0_attention_self_q_bias = constants_->at(4); 2024-04-20T23:43:33.3111436Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3115539Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1928:10: warning: variable ‘deberta_encoder_layer_0_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3118461Z 1928 | auto deberta_encoder_layer_0_attention_self_v_bias = constants_->at(5); 2024-04-20T23:43:33.3120108Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3122699Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1940:10: warning: variable ‘deberta_encoder_layer_1_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3125167Z 1940 | auto deberta_encoder_layer_1_attention_self_q_bias = constants_->at(17); 2024-04-20T23:43:33.3125966Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3128418Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1941:10: warning: variable ‘deberta_encoder_layer_1_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3131026Z 1941 | auto deberta_encoder_layer_1_attention_self_v_bias = constants_->at(18); 2024-04-20T23:43:33.3131852Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3134700Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1953:10: warning: variable ‘deberta_encoder_layer_2_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3137214Z 1953 | auto deberta_encoder_layer_2_attention_self_q_bias = constants_->at(30); 2024-04-20T23:43:33.3138041Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3140442Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1954:10: warning: variable ‘deberta_encoder_layer_2_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3142864Z 1954 | auto deberta_encoder_layer_2_attention_self_v_bias = constants_->at(31); 2024-04-20T23:43:33.3143876Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3146626Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1966:10: warning: variable ‘deberta_encoder_layer_3_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3149059Z 1966 | auto deberta_encoder_layer_3_attention_self_q_bias = constants_->at(43); 2024-04-20T23:43:33.3149886Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3152277Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1967:10: warning: variable ‘deberta_encoder_layer_3_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3154700Z 1967 | auto deberta_encoder_layer_3_attention_self_v_bias = constants_->at(44); 2024-04-20T23:43:33.3155496Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3157974Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1979:10: warning: variable ‘deberta_encoder_layer_4_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3160386Z 1979 | auto deberta_encoder_layer_4_attention_self_q_bias = constants_->at(56); 2024-04-20T23:43:33.3161199Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3163589Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1980:10: warning: variable ‘deberta_encoder_layer_4_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3166002Z 1980 | auto deberta_encoder_layer_4_attention_self_v_bias = constants_->at(57); 2024-04-20T23:43:33.3166843Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3169257Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1992:10: warning: variable ‘deberta_encoder_layer_5_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3172074Z 1992 | auto deberta_encoder_layer_5_attention_self_q_bias = constants_->at(69); 2024-04-20T23:43:33.3173051Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3175446Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:1993:10: warning: variable ‘deberta_encoder_layer_5_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3177905Z 1993 | auto deberta_encoder_layer_5_attention_self_v_bias = constants_->at(70); 2024-04-20T23:43:33.3178718Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3181109Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2005:10: warning: variable ‘deberta_encoder_layer_6_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3183541Z 2005 | auto deberta_encoder_layer_6_attention_self_q_bias = constants_->at(82); 2024-04-20T23:43:33.3184511Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3187165Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2006:10: warning: variable ‘deberta_encoder_layer_6_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3189579Z 2006 | auto deberta_encoder_layer_6_attention_self_v_bias = constants_->at(83); 2024-04-20T23:43:33.3190394Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3192787Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2018:10: warning: variable ‘deberta_encoder_layer_7_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3195427Z 2018 | auto deberta_encoder_layer_7_attention_self_q_bias = constants_->at(95); 2024-04-20T23:43:33.3196248Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3198688Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2019:10: warning: variable ‘deberta_encoder_layer_7_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3201114Z 2019 | auto deberta_encoder_layer_7_attention_self_v_bias = constants_->at(96); 2024-04-20T23:43:33.3201933Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3204306Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2031:10: warning: variable ‘deberta_encoder_layer_8_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3206764Z 2031 | auto deberta_encoder_layer_8_attention_self_q_bias = constants_->at(108); 2024-04-20T23:43:33.3207585Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3209973Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2032:10: warning: variable ‘deberta_encoder_layer_8_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3212370Z 2032 | auto deberta_encoder_layer_8_attention_self_v_bias = constants_->at(109); 2024-04-20T23:43:33.3213448Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3215872Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2044:10: warning: variable ‘deberta_encoder_layer_9_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3218568Z 2044 | auto deberta_encoder_layer_9_attention_self_q_bias = constants_->at(121); 2024-04-20T23:43:33.3219396Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3221975Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2045:10: warning: variable ‘deberta_encoder_layer_9_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3224806Z 2045 | auto deberta_encoder_layer_9_attention_self_v_bias = constants_->at(122); 2024-04-20T23:43:33.3225749Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3228202Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2057:10: warning: variable ‘deberta_encoder_layer_10_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3230666Z 2057 | auto deberta_encoder_layer_10_attention_self_q_bias = constants_->at(134); 2024-04-20T23:43:33.3231489Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3233895Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2058:10: warning: variable ‘deberta_encoder_layer_10_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3236310Z 2058 | auto deberta_encoder_layer_10_attention_self_v_bias = constants_->at(135); 2024-04-20T23:43:33.3237164Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3239562Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2070:10: warning: variable ‘deberta_encoder_layer_11_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3242006Z 2070 | auto deberta_encoder_layer_11_attention_self_q_bias = constants_->at(147); 2024-04-20T23:43:33.3242826Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:33.3245389Z /tmp/torchinductor_jenkins/c4sxcz66bx4ksbbzywe5yi7gblgb5e7kljb2u6rzfqnwmgdtifok/ctcpuzoa3r2e55tsqremux5rohqdlvpm4vbrzaxmspxpv5qg7ulp.cpp:2071:10: warning: variable ‘deberta_encoder_layer_11_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:43:33.3247881Z 2071 | auto deberta_encoder_layer_11_attention_self_v_bias = constants_->at(148); 2024-04-20T23:43:33.3248706Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:43:39.9881911Z pass 2024-04-20T23:43:46.4371283Z 2024-04-20T23:43:49.4955815Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:43:49.4956448Z loading model: 0it [00:03, ?it/s] 2024-04-20T23:43:49.5000646Z cuda eval DebertaForQuestionAnswering 2024-04-20T23:44:29.0474236Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-04-20T23:44:29.0479946Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1898:10: warning: variable ‘deberta_encoder_layer_0_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0482388Z 1898 | auto deberta_encoder_layer_0_attention_self_q_bias = constants_->at(4); 2024-04-20T23:44:29.0483203Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0485675Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1899:10: warning: variable ‘deberta_encoder_layer_0_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0488593Z 1899 | auto deberta_encoder_layer_0_attention_self_v_bias = constants_->at(5); 2024-04-20T23:44:29.0489416Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0491819Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1911:10: warning: variable ‘deberta_encoder_layer_1_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0494568Z 1911 | auto deberta_encoder_layer_1_attention_self_q_bias = constants_->at(17); 2024-04-20T23:44:29.0495380Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0497832Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1912:10: warning: variable ‘deberta_encoder_layer_1_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0500273Z 1912 | auto deberta_encoder_layer_1_attention_self_v_bias = constants_->at(18); 2024-04-20T23:44:29.0501088Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0503470Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1924:10: warning: variable ‘deberta_encoder_layer_2_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0505928Z 1924 | auto deberta_encoder_layer_2_attention_self_q_bias = constants_->at(30); 2024-04-20T23:44:29.0506816Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0509223Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1925:10: warning: variable ‘deberta_encoder_layer_2_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0511646Z 1925 | auto deberta_encoder_layer_2_attention_self_v_bias = constants_->at(31); 2024-04-20T23:44:29.0512439Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0515103Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1937:10: warning: variable ‘deberta_encoder_layer_3_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0517596Z 1937 | auto deberta_encoder_layer_3_attention_self_q_bias = constants_->at(43); 2024-04-20T23:44:29.0518403Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0520794Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1938:10: warning: variable ‘deberta_encoder_layer_3_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0523196Z 1938 | auto deberta_encoder_layer_3_attention_self_v_bias = constants_->at(44); 2024-04-20T23:44:29.0524014Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0526445Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1950:10: warning: variable ‘deberta_encoder_layer_4_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0528863Z 1950 | auto deberta_encoder_layer_4_attention_self_q_bias = constants_->at(56); 2024-04-20T23:44:29.0529673Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0532040Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1951:10: warning: variable ‘deberta_encoder_layer_4_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0534739Z 1951 | auto deberta_encoder_layer_4_attention_self_v_bias = constants_->at(57); 2024-04-20T23:44:29.0535737Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0538193Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1963:10: warning: variable ‘deberta_encoder_layer_5_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0540609Z 1963 | auto deberta_encoder_layer_5_attention_self_q_bias = constants_->at(69); 2024-04-20T23:44:29.0541407Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0543864Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1964:10: warning: variable ‘deberta_encoder_layer_5_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0546311Z 1964 | auto deberta_encoder_layer_5_attention_self_v_bias = constants_->at(70); 2024-04-20T23:44:29.0547136Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0549548Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1976:10: warning: variable ‘deberta_encoder_layer_6_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0551972Z 1976 | auto deberta_encoder_layer_6_attention_self_q_bias = constants_->at(82); 2024-04-20T23:44:29.0552769Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0555163Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1977:10: warning: variable ‘deberta_encoder_layer_6_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0557603Z 1977 | auto deberta_encoder_layer_6_attention_self_v_bias = constants_->at(83); 2024-04-20T23:44:29.0558413Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0560951Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1989:10: warning: variable ‘deberta_encoder_layer_7_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0563386Z 1989 | auto deberta_encoder_layer_7_attention_self_q_bias = constants_->at(95); 2024-04-20T23:44:29.0564201Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0566629Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:1990:10: warning: variable ‘deberta_encoder_layer_7_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0569037Z 1990 | auto deberta_encoder_layer_7_attention_self_v_bias = constants_->at(96); 2024-04-20T23:44:29.0569830Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0572247Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2002:10: warning: variable ‘deberta_encoder_layer_8_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0574936Z 2002 | auto deberta_encoder_layer_8_attention_self_q_bias = constants_->at(108); 2024-04-20T23:44:29.0575753Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0578210Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2003:10: warning: variable ‘deberta_encoder_layer_8_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0580639Z 2003 | auto deberta_encoder_layer_8_attention_self_v_bias = constants_->at(109); 2024-04-20T23:44:29.0581433Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0583827Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2015:10: warning: variable ‘deberta_encoder_layer_9_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0586467Z 2015 | auto deberta_encoder_layer_9_attention_self_q_bias = constants_->at(121); 2024-04-20T23:44:29.0587282Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0589673Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2016:10: warning: variable ‘deberta_encoder_layer_9_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0592061Z 2016 | auto deberta_encoder_layer_9_attention_self_v_bias = constants_->at(122); 2024-04-20T23:44:29.0592871Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0595270Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2028:10: warning: variable ‘deberta_encoder_layer_10_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0597755Z 2028 | auto deberta_encoder_layer_10_attention_self_q_bias = constants_->at(134); 2024-04-20T23:44:29.0598574Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0600963Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2029:10: warning: variable ‘deberta_encoder_layer_10_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0603383Z 2029 | auto deberta_encoder_layer_10_attention_self_v_bias = constants_->at(135); 2024-04-20T23:44:29.0604203Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0606635Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2041:10: warning: variable ‘deberta_encoder_layer_11_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0609233Z 2041 | auto deberta_encoder_layer_11_attention_self_q_bias = constants_->at(147); 2024-04-20T23:44:29.0610052Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:29.0612455Z /tmp/torchinductor_jenkins/cftkttadvufujir2s7gt5odu7lfqkxsjjpdpbh4vk2r2rf6p3cs4/c6yqunjtkeo76cno3swb3io4squcjfhxhppl45wnui7ej2nf5hqr.cpp:2042:10: warning: variable ‘deberta_encoder_layer_11_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-20T23:44:29.0615153Z 2042 | auto deberta_encoder_layer_11_attention_self_v_bias = constants_->at(148); 2024-04-20T23:44:29.0615982Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-20T23:44:35.6325751Z pass 2024-04-20T23:44:41.1371355Z 2024-04-20T23:44:58.6873114Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:44:58.6874045Z loading model: 0it [00:17, ?it/s] 2024-04-20T23:44:58.6990836Z cuda eval DebertaV2ForMaskedLM 2024-04-20T23:44:58.6997783Z pass_due_to_skip 2024-04-20T23:45:03.4642576Z 2024-04-20T23:45:18.2403615Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:45:18.2405074Z loading model: 0it [00:14, ?it/s] 2024-04-20T23:45:18.2513094Z cuda eval DebertaV2ForQuestionAnswering 2024-04-20T23:47:12.7929367Z pass 2024-04-20T23:47:19.2314410Z 2024-04-20T23:47:21.6998041Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:47:21.6998623Z loading model: 0it [00:02, ?it/s] 2024-04-20T23:47:21.7029832Z cuda eval DistilBertForMaskedLM 2024-04-20T23:47:55.8021841Z pass 2024-04-20T23:47:57.2455423Z accuracy pass_rate=81.25% 2024-04-20T23:47:57.2456071Z calls_captured gmean=0.00x mean=453.875x 2024-04-20T23:47:57.2458215Z unique_graphs gmean=0.00x mean=0.812x 2024-04-20T23:47:57.2461150Z graph_breaks gmean=0.00x mean=0.000x 2024-04-20T23:47:57.2463768Z unique_graph_breaks gmean=0.00x mean=0.000x 2024-04-20T23:47:57.2465734Z autograd_captures gmean=0.00x mean=0.000x 2024-04-20T23:47:57.2468513Z autograd_compiles gmean=0.00x mean=0.000x 2024-04-20T23:47:57.9195595Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *maxautotune-true* ]] 2024-04-20T23:47:57.9198995Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *cudagraphs_low_precision-true* ]] 2024-04-20T23:47:57.9200909Z + [[ inference == \i\n\f\e\r\e\n\c\e ]] 2024-04-20T23:47:57.9203606Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --inference --quant --backend inductor --device cuda --total-partitions 3 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_cudagraphs_low_precision_huggingface_quant_inference_cuda_accuracy.csv 2024-04-20T23:48:02.3438707Z usage: huggingface.py 2024-04-20T23:48:02.3439479Z [-h] 2024-04-20T23:48:02.3439884Z [--filter FILTER] 2024-04-20T23:48:02.3440346Z [--exclude EXCLUDE] 2024-04-20T23:48:02.3440860Z [--exclude-exact EXCLUDE_EXACT] 2024-04-20T23:48:02.3441510Z [--total-partitions {1,2,3,4,5,6,7,8,9}] 2024-04-20T23:48:02.3442297Z [--partition-id PARTITION_ID] 2024-04-20T23:48:02.3448038Z [--devices DEVICES] 2024-04-20T23:48:02.3448675Z [--device-index DEVICE_INDEX] 2024-04-20T23:48:02.3449239Z [--repeat REPEAT] 2024-04-20T23:48:02.3449790Z [--iterations-per-run ITERATIONS_PER_RUN] 2024-04-20T23:48:02.3450404Z [--randomize-input] 2024-04-20T23:48:02.3450874Z [--threads THREADS] 2024-04-20T23:48:02.3451378Z [--nopython] 2024-04-20T23:48:02.3451783Z [--no-skip] 2024-04-20T23:48:02.3452192Z [--prims-nvfuser] 2024-04-20T23:48:02.3452659Z [--dump-raw-metrics] 2024-04-20T23:48:02.3453385Z [--log-operator-inputs] 2024-04-20T23:48:02.3454163Z [--channels-last] 2024-04-20T23:48:02.3454957Z [--batch-size BATCH_SIZE] 2024-04-20T23:48:02.3455872Z [--iterations ITERATIONS] 2024-04-20T23:48:02.3456919Z [--batch-size-file BATCH_SIZE_FILE] 2024-04-20T23:48:02.3458008Z [--cosine] 2024-04-20T23:48:02.3458418Z [--freezing] 2024-04-20T23:48:02.3458803Z [--ci] 2024-04-20T23:48:02.3459163Z [--dashboard] 2024-04-20T23:48:02.3459596Z [--skip-fp64-check] 2024-04-20T23:48:02.3460036Z [--fast] 2024-04-20T23:48:02.3460402Z [--only ONLY] 2024-04-20T23:48:02.3460818Z [--multiprocess] 2024-04-20T23:48:02.3461232Z [--ddp] 2024-04-20T23:48:02.3461674Z [--fsdp] 2024-04-20T23:48:02.3462136Z [--optimize-ddp-mode OPTIMIZE_DDP_MODE] 2024-04-20T23:48:02.3462874Z [--distributed-master-port DISTRIBUTED_MASTER_PORT] 2024-04-20T23:48:02.3463527Z [--dynamic-shapes] 2024-04-20T23:48:02.3464008Z [--dynamic-batch-only] 2024-04-20T23:48:02.3464492Z [--specialize-int] 2024-04-20T23:48:02.3464942Z [--use-eval-mode] 2024-04-20T23:48:02.3465412Z [--skip-accuracy-check] 2024-04-20T23:48:02.3465938Z [--generate-aot-autograd-stats] 2024-04-20T23:48:02.3466508Z [--inductor-settings] 2024-04-20T23:48:02.3466988Z [--suppress-errors] 2024-04-20T23:48:02.3467445Z [--output OUTPUT] 2024-04-20T23:48:02.3467957Z [--output-directory OUTPUT_DIRECTORY] 2024-04-20T23:48:02.3468537Z [--baseline BASELINE] 2024-04-20T23:48:02.3468997Z [--part PART] 2024-04-20T23:48:02.3469445Z [--export-profiler-trace] 2024-04-20T23:48:02.3470041Z [--profiler-trace-name PROFILER_TRACE_NAME] 2024-04-20T23:48:02.3470674Z [--diff-branch DIFF_BRANCH] 2024-04-20T23:48:02.3471157Z [--tag TAG] 2024-04-20T23:48:02.3471574Z [--explain] 2024-04-20T23:48:02.3471964Z [--stats] 2024-04-20T23:48:02.3472383Z [--use-warm-peak-memory] 2024-04-20T23:48:02.3472874Z [--print-memory] 2024-04-20T23:48:02.3473563Z [--print-compilation-time] 2024-04-20T23:48:02.3474117Z [--print-dataframe-summary] 2024-04-20T23:48:02.3474651Z [--cold-start-latency] 2024-04-20T23:48:02.3475151Z [--disable-cudagraphs] 2024-04-20T23:48:02.3475661Z [--disable-split-reductions] 2024-04-20T23:48:02.3476259Z [--disable-persistent-reductions] 2024-04-20T23:48:02.3476860Z [--disable-divisible-by-16] 2024-04-20T23:48:02.3477489Z [--inductor-compile-mode INDUCTOR_COMPILE_MODE] 2024-04-20T23:48:02.3478124Z [--print-graph-breaks] 2024-04-20T23:48:02.3478616Z [--log-graph-breaks] 2024-04-20T23:48:02.3479082Z [--trace-on-xla] 2024-04-20T23:48:02.3479545Z [--xla-tolerance XLA_TOLERANCE] 2024-04-20T23:48:02.3480088Z [--collect-outputs] 2024-04-20T23:48:02.3480622Z [--enable-activation-checkpointing] 2024-04-20T23:48:02.3481180Z [--timing] 2024-04-20T23:48:02.3481736Z [--progress] 2024-04-20T23:48:02.3482160Z [--timeout TIMEOUT] 2024-04-20T23:48:02.3482819Z [--per_process_memory_fraction PER_PROCESS_MEMORY_FRACTION] 2024-04-20T23:48:02.3483547Z [--no-translation-validation] 2024-04-20T23:48:02.3484055Z [--minify] 2024-04-20T23:48:02.3484477Z [--compiled-autograd] 2024-04-20T23:48:02.3484990Z [--profile_dynamo_cache_lookup] 2024-04-20T23:48:02.3485520Z [--snapshot-memory] 2024-04-20T23:48:02.3485959Z [--nnc] 2024-04-20T23:48:02.3486439Z [--float16 | --bfloat16 | --float32 | --amp] 2024-04-20T23:48:02.3487075Z [--amp-dtype {bfloat16,float16}] 2024-04-20T23:48:02.3487610Z [--verbose | --quiet] 2024-04-20T23:48:02.3493124Z [--coverage | --overhead | --speedup-dynamo-ts | --speedup-fx2trt | --speedup-fx2trt-fp16 | --print-fx | --print-aten-ops | --inductor | --export | --export-aot-inductor | --xla | --torchscript-onnx | --dynamo-onnx | --dynamo-onnx-aot-inline | --backend {aot_eager,aot_eager_decomp_partition,aot_eager_default_partitioner,aot_ts,cudagraphs,dynamo_accuracy_minifier_backend,dynamo_minifier_backend,eager,eager_debug,inductor,non_leaf_compile_error_TESTING_ONLY,onnxrt,openxla,openxla_eval,pre_dispatch_eager,relu_accuracy_error_TESTING_ONLY,relu_compile_error_TESTING_ONLY,relu_runtime_error_TESTING_ONLY,ts,tvm} | --nothing | --log-conv-args | --recompile-profiler | --find-batch-sizes] 2024-04-20T23:48:02.3498216Z (--accuracy | --performance | --tolerance) 2024-04-20T23:48:02.3498837Z (--training | --inference) 2024-04-20T23:48:02.3499505Z huggingface.py: error: unrecognized arguments: --quant 2024-04-20T23:48:03.0254069Z + true 2024-04-20T23:48:03.0256658Z + cp /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_huggingface_bfloat16_inference_cuda_accuracy.csv /var/lib/jenkins/workspace/test/test-reports/inductor_cudagraphs_low_precision_huggingface_quant_inference_cuda_accuracy.csv 2024-04-20T23:48:03.0283137Z + for target in "${targets[@]}" 2024-04-20T23:48:03.0284182Z + target_flag=("--${target}") 2024-04-20T23:48:03.0284983Z + local target_flag 2024-04-20T23:48:03.0285657Z + [[ performance == \p\e\r\f\o\r\m\a\n\c\e ]] 2024-04-20T23:48:03.0286420Z + target_flag+=(--cold-start-latency) 2024-04-20T23:48:03.0288780Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true-cudagraphs_low_precision-true == *default-true* ]] 2024-04-20T23:48:03.0293152Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_bfloat16_inference_cuda_performance.csv 2024-04-20T23:48:11.8270780Z 2024-04-20T23:48:16.3957563Z loading model: 0it [00:00, ?it/s] 2024-04-20T23:48:16.3958469Z loading model: 0it [00:04, ?it/s] 2024-04-20T23:48:16.3964745Z cuda eval AlbertForMaskedLM 2024-04-20T23:48:34.6802832Z 2024-04-20T23:48:34.9480310Z running benchmark: 0% 0/30 [00:00at(4); 2024-04-21T00:52:34.1438263Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1440636Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1932:10: warning: variable ‘deberta_encoder_layer_0_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1443559Z 1932 | auto deberta_encoder_layer_0_attention_self_v_bias = constants_->at(5); 2024-04-21T00:52:34.1444383Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1446779Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1944:10: warning: variable ‘deberta_encoder_layer_1_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1449170Z 1944 | auto deberta_encoder_layer_1_attention_self_q_bias = constants_->at(17); 2024-04-21T00:52:34.1449994Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1452363Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1945:10: warning: variable ‘deberta_encoder_layer_1_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1455084Z 1945 | auto deberta_encoder_layer_1_attention_self_v_bias = constants_->at(18); 2024-04-21T00:52:34.1455892Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1458307Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1957:10: warning: variable ‘deberta_encoder_layer_2_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1460697Z 1957 | auto deberta_encoder_layer_2_attention_self_q_bias = constants_->at(30); 2024-04-21T00:52:34.1461505Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1463864Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1958:10: warning: variable ‘deberta_encoder_layer_2_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1466277Z 1958 | auto deberta_encoder_layer_2_attention_self_v_bias = constants_->at(31); 2024-04-21T00:52:34.1467099Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1469706Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1970:10: warning: variable ‘deberta_encoder_layer_3_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1472104Z 1970 | auto deberta_encoder_layer_3_attention_self_q_bias = constants_->at(43); 2024-04-21T00:52:34.1472914Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1475255Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1971:10: warning: variable ‘deberta_encoder_layer_3_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1477791Z 1971 | auto deberta_encoder_layer_3_attention_self_v_bias = constants_->at(44); 2024-04-21T00:52:34.1478607Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1480953Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1983:10: warning: variable ‘deberta_encoder_layer_4_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1483338Z 1983 | auto deberta_encoder_layer_4_attention_self_q_bias = constants_->at(56); 2024-04-21T00:52:34.1484138Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1486533Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1984:10: warning: variable ‘deberta_encoder_layer_4_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1488920Z 1984 | auto deberta_encoder_layer_4_attention_self_v_bias = constants_->at(57); 2024-04-21T00:52:34.1489725Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1492276Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1996:10: warning: variable ‘deberta_encoder_layer_5_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1495050Z 1996 | auto deberta_encoder_layer_5_attention_self_q_bias = constants_->at(69); 2024-04-21T00:52:34.1495868Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1498269Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:1997:10: warning: variable ‘deberta_encoder_layer_5_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1500663Z 1997 | auto deberta_encoder_layer_5_attention_self_v_bias = constants_->at(70); 2024-04-21T00:52:34.1501476Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1503835Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2009:10: warning: variable ‘deberta_encoder_layer_6_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1506299Z 2009 | auto deberta_encoder_layer_6_attention_self_q_bias = constants_->at(82); 2024-04-21T00:52:34.1507107Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1509462Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2010:10: warning: variable ‘deberta_encoder_layer_6_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1511833Z 2010 | auto deberta_encoder_layer_6_attention_self_v_bias = constants_->at(83); 2024-04-21T00:52:34.1512631Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1514986Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2022:10: warning: variable ‘deberta_encoder_layer_7_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1517655Z 2022 | auto deberta_encoder_layer_7_attention_self_q_bias = constants_->at(95); 2024-04-21T00:52:34.1518472Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1520835Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2023:10: warning: variable ‘deberta_encoder_layer_7_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1523221Z 2023 | auto deberta_encoder_layer_7_attention_self_v_bias = constants_->at(96); 2024-04-21T00:52:34.1524013Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1526399Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2035:10: warning: variable ‘deberta_encoder_layer_8_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1528797Z 2035 | auto deberta_encoder_layer_8_attention_self_q_bias = constants_->at(108); 2024-04-21T00:52:34.1529614Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1531955Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2036:10: warning: variable ‘deberta_encoder_layer_8_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1534638Z 2036 | auto deberta_encoder_layer_8_attention_self_v_bias = constants_->at(109); 2024-04-21T00:52:34.1535453Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1537856Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2048:10: warning: variable ‘deberta_encoder_layer_9_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1540462Z 2048 | auto deberta_encoder_layer_9_attention_self_q_bias = constants_->at(121); 2024-04-21T00:52:34.1541270Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1543625Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2049:10: warning: variable ‘deberta_encoder_layer_9_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1546010Z 2049 | auto deberta_encoder_layer_9_attention_self_v_bias = constants_->at(122); 2024-04-21T00:52:34.1546863Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1549222Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2061:10: warning: variable ‘deberta_encoder_layer_10_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1551645Z 2061 | auto deberta_encoder_layer_10_attention_self_q_bias = constants_->at(134); 2024-04-21T00:52:34.1552456Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1554825Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2062:10: warning: variable ‘deberta_encoder_layer_10_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1557248Z 2062 | auto deberta_encoder_layer_10_attention_self_v_bias = constants_->at(135); 2024-04-21T00:52:34.1558065Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1560402Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2074:10: warning: variable ‘deberta_encoder_layer_11_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1562818Z 2074 | auto deberta_encoder_layer_11_attention_self_q_bias = constants_->at(147); 2024-04-21T00:52:34.1563647Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:34.1566207Z /tmp/tmpj8zqah6w/cgaate32v4tcddc5ytkbjer3ghxbip4mr3eks2irx2dw2shlhpvk/caiccim3s2ialf6uewfvsyxmk6dq3rk2i2s2widctzxd5anf5rbt.cpp:2075:10: warning: variable ‘deberta_encoder_layer_11_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:52:34.1568638Z 2075 | auto deberta_encoder_layer_11_attention_self_v_bias = constants_->at(148); 2024-04-21T00:52:34.1569465Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:52:40.7831220Z 2024-04-21T00:52:40.9055482Z running benchmark: 0% 0/30 [00:00at(4); 2024-04-21T00:53:39.4616697Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4619080Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1901:10: warning: variable ‘deberta_encoder_layer_0_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4621490Z 1901 | auto deberta_encoder_layer_0_attention_self_v_bias = constants_->at(5); 2024-04-21T00:53:39.4622294Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4624662Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1913:10: warning: variable ‘deberta_encoder_layer_1_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4627275Z 1913 | auto deberta_encoder_layer_1_attention_self_q_bias = constants_->at(17); 2024-04-21T00:53:39.4628095Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4630452Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1914:10: warning: variable ‘deberta_encoder_layer_1_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4632849Z 1914 | auto deberta_encoder_layer_1_attention_self_v_bias = constants_->at(18); 2024-04-21T00:53:39.4633667Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4636350Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1926:10: warning: variable ‘deberta_encoder_layer_2_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4638782Z 1926 | auto deberta_encoder_layer_2_attention_self_q_bias = constants_->at(30); 2024-04-21T00:53:39.4639572Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4641950Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1927:10: warning: variable ‘deberta_encoder_layer_2_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4644345Z 1927 | auto deberta_encoder_layer_2_attention_self_v_bias = constants_->at(31); 2024-04-21T00:53:39.4645158Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4647573Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1939:10: warning: variable ‘deberta_encoder_layer_3_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4649966Z 1939 | auto deberta_encoder_layer_3_attention_self_q_bias = constants_->at(43); 2024-04-21T00:53:39.4650765Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4653481Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1940:10: warning: variable ‘deberta_encoder_layer_3_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4655925Z 1940 | auto deberta_encoder_layer_3_attention_self_v_bias = constants_->at(44); 2024-04-21T00:53:39.4656788Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4659354Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1952:10: warning: variable ‘deberta_encoder_layer_4_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4661790Z 1952 | auto deberta_encoder_layer_4_attention_self_q_bias = constants_->at(56); 2024-04-21T00:53:39.4662609Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4664969Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1953:10: warning: variable ‘deberta_encoder_layer_4_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4667401Z 1953 | auto deberta_encoder_layer_4_attention_self_v_bias = constants_->at(57); 2024-04-21T00:53:39.4668197Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4670572Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1965:10: warning: variable ‘deberta_encoder_layer_5_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4672982Z 1965 | auto deberta_encoder_layer_5_attention_self_q_bias = constants_->at(69); 2024-04-21T00:53:39.4673790Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4676152Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1966:10: warning: variable ‘deberta_encoder_layer_5_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4678579Z 1966 | auto deberta_encoder_layer_5_attention_self_v_bias = constants_->at(70); 2024-04-21T00:53:39.4679373Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4681745Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1978:10: warning: variable ‘deberta_encoder_layer_6_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4684310Z 1978 | auto deberta_encoder_layer_6_attention_self_q_bias = constants_->at(82); 2024-04-21T00:53:39.4685124Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4687506Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1979:10: warning: variable ‘deberta_encoder_layer_6_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4689904Z 1979 | auto deberta_encoder_layer_6_attention_self_v_bias = constants_->at(83); 2024-04-21T00:53:39.4690711Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4693372Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1991:10: warning: variable ‘deberta_encoder_layer_7_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4695885Z 1991 | auto deberta_encoder_layer_7_attention_self_q_bias = constants_->at(95); 2024-04-21T00:53:39.4696763Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4699119Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:1992:10: warning: variable ‘deberta_encoder_layer_7_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4701512Z 1992 | auto deberta_encoder_layer_7_attention_self_v_bias = constants_->at(96); 2024-04-21T00:53:39.4702318Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4704681Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2004:10: warning: variable ‘deberta_encoder_layer_8_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4707324Z 2004 | auto deberta_encoder_layer_8_attention_self_q_bias = constants_->at(108); 2024-04-21T00:53:39.4708128Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4710496Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2005:10: warning: variable ‘deberta_encoder_layer_8_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4712885Z 2005 | auto deberta_encoder_layer_8_attention_self_v_bias = constants_->at(109); 2024-04-21T00:53:39.4713707Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4716064Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2017:10: warning: variable ‘deberta_encoder_layer_9_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4718487Z 2017 | auto deberta_encoder_layer_9_attention_self_q_bias = constants_->at(121); 2024-04-21T00:53:39.4719296Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4721677Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2018:10: warning: variable ‘deberta_encoder_layer_9_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4724073Z 2018 | auto deberta_encoder_layer_9_attention_self_v_bias = constants_->at(122); 2024-04-21T00:53:39.4724884Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4727273Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2030:10: warning: variable ‘deberta_encoder_layer_10_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4729687Z 2030 | auto deberta_encoder_layer_10_attention_self_q_bias = constants_->at(134); 2024-04-21T00:53:39.4730517Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4733254Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2031:10: warning: variable ‘deberta_encoder_layer_10_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4735729Z 2031 | auto deberta_encoder_layer_10_attention_self_v_bias = constants_->at(135); 2024-04-21T00:53:39.4736589Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4738971Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2043:10: warning: variable ‘deberta_encoder_layer_11_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4741384Z 2043 | auto deberta_encoder_layer_11_attention_self_q_bias = constants_->at(147); 2024-04-21T00:53:39.4742218Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:39.4744603Z /tmp/tmp2dmbcxuw/cgiqqahc5l5sjb5o7knos5enchqhdvkmo2riblxnd5cwptv525dx/cm6po7empkcklkhfjjugjmh4ehejjn7rshdgdlekm6lt5lpjpmct.cpp:2044:10: warning: variable ‘deberta_encoder_layer_11_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-04-21T00:53:39.4747045Z 2044 | auto deberta_encoder_layer_11_attention_self_v_bias = constants_->at(148); 2024-04-21T00:53:39.4747867Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-21T00:53:46.0770881Z 2024-04-21T00:53:46.1872441Z running benchmark: 0% 0/30 [00:00