2024-06-26T06:59:43.2736416Z Current runner version: '2.317.0' 2024-06-26T06:59:43.2742515Z Runner name: 'i-0467b1cafd1897be0' 2024-06-26T06:59:43.2743392Z Runner group name: 'Default' 2024-06-26T06:59:43.2744372Z Machine name: 'ip-10-0-15-83' 2024-06-26T06:59:43.2749277Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T06:59:43.2751233Z Actions: read 2024-06-26T06:59:43.2751879Z Attestations: read 2024-06-26T06:59:43.2752434Z Checks: read 2024-06-26T06:59:43.2752922Z Contents: read 2024-06-26T06:59:43.2753483Z Deployments: read 2024-06-26T06:59:43.2754208Z Discussions: read 2024-06-26T06:59:43.2754759Z Issues: read 2024-06-26T06:59:43.2755319Z Metadata: read 2024-06-26T06:59:43.2755847Z Packages: read 2024-06-26T06:59:43.2756376Z Pages: read 2024-06-26T06:59:43.2756915Z PullRequests: read 2024-06-26T06:59:43.2757486Z RepositoryProjects: read 2024-06-26T06:59:43.2758171Z SecurityEvents: read 2024-06-26T06:59:43.2758765Z Statuses: read 2024-06-26T06:59:43.2759280Z ##[endgroup] 2024-06-26T06:59:43.2762702Z Secret source: Actions 2024-06-26T06:59:43.2763542Z Prepare workflow directory 2024-06-26T06:59:43.6345969Z Prepare all required actions 2024-06-26T06:59:43.6530833Z Getting action download info 2024-06-26T06:59:43.8166638Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T06:59:44.1463815Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T06:59:46.9666110Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T06:59:47.0952043Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T06:59:47.3576125Z Getting action download info 2024-06-26T06:59:47.4654341Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T06:59:47.6317500Z Getting action download info 2024-06-26T06:59:47.7533068Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T06:59:47.9145394Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T06:59:47.9147729Z ##[group] Inputs 2024-06-26T06:59:47.9148342Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T06:59:47.9157443Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T06:59:47.9167222Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:59:47.9168497Z sync-tag: 2024-06-26T06:59:47.9169508Z timeout-minutes: 240 2024-06-26T06:59:47.9169925Z use-gha: 2024-06-26T06:59:47.9170279Z dashboard-tag: 2024-06-26T06:59:47.9170680Z s3-bucket: gha-artifacts 2024-06-26T06:59:47.9171117Z aws-role-to-assume: 2024-06-26T06:59:47.9171528Z ##[endgroup] 2024-06-26T06:59:47.9172512Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T06:59:47.9756109Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T06:59:47.9898675Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T06:59:47.9909278Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:59:47.9909888Z ##[endgroup] 2024-06-26T06:59:48.9313642Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T06:59:48.9314522Z with: 2024-06-26T06:59:48.9315321Z github-secret: *** 2024-06-26T06:59:48.9316355Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T06:59:48.9317446Z activate-with-label: false 2024-06-26T06:59:48.9317894Z label: with-ssh 2024-06-26T06:59:48.9318316Z remove-existing-keys: true 2024-06-26T06:59:48.9318787Z fail-silently: true 2024-06-26T06:59:48.9319186Z env: 2024-06-26T06:59:48.9319568Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:48.9319993Z ##[endgroup] 2024-06-26T06:59:49.0189032Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T06:59:49.0190170Z ciflow reference detected, attempting to extract PR number 2024-06-26T06:59:49.3355211Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T06:59:49.4015626Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T06:59:49.4016446Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T06:59:49.4610797Z No SSH keys found for user leslie-fang-intel 2024-06-26T06:59:49.4725414Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T06:59:49.4726128Z with: 2024-06-26T06:59:49.4726521Z submodules: recursive 2024-06-26T06:59:49.4727018Z fetch-depth: 0 2024-06-26T06:59:49.4727413Z env: 2024-06-26T06:59:49.4727797Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:49.4728287Z ##[endgroup] 2024-06-26T06:59:49.4964005Z ##[group]Run retry () { 2024-06-26T06:59:49.4964670Z retry () { 2024-06-26T06:59:49.4965275Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T06:59:49.4965923Z } 2024-06-26T06:59:49.4966323Z echo "${GITHUB_WORKSPACE}" 2024-06-26T06:59:49.4966837Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T06:59:49.4967410Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:59:49.4967957Z else 2024-06-26T06:59:49.4968393Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:59:49.4968917Z fi 2024-06-26T06:59:49.4969343Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T06:59:49.4977532Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:59:49.4978116Z env: 2024-06-26T06:59:49.4978477Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:49.4978907Z NO_SUDO: 2024-06-26T06:59:49.4979261Z ##[endgroup] 2024-06-26T06:59:49.5004014Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:59:51.7466731Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T06:59:51.7467245Z with: 2024-06-26T06:59:51.7467648Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:59:51.7468168Z fetch-depth: 0 2024-06-26T06:59:51.7468564Z submodules: recursive 2024-06-26T06:59:51.7468981Z quiet-checkout: true 2024-06-26T06:59:51.7469416Z repository: pytorch/pytorch 2024-06-26T06:59:51.7469984Z token: *** 2024-06-26T06:59:51.7470361Z ssh-strict: true 2024-06-26T06:59:51.7470771Z persist-credentials: true 2024-06-26T06:59:51.7471217Z clean: true 2024-06-26T06:59:51.7471639Z sparse-checkout-cone-mode: true 2024-06-26T06:59:51.7472113Z lfs: false 2024-06-26T06:59:51.7472497Z set-safe-directory: true 2024-06-26T06:59:51.7472921Z env: 2024-06-26T06:59:51.7473277Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:51.7473694Z ##[endgroup] 2024-06-26T06:59:51.8401950Z Syncing repository: pytorch/pytorch 2024-06-26T06:59:51.8403691Z ##[group]Getting Git version info 2024-06-26T06:59:51.8404706Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:59:51.8405697Z [command]/usr/bin/git version 2024-06-26T06:59:51.8406152Z git version 2.40.1 2024-06-26T06:59:51.8408107Z ##[endgroup] 2024-06-26T06:59:51.8419849Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/68fbae9e-63f4-41d4-a153-73d5ee0169cb' before making global git config changes 2024-06-26T06:59:51.8421191Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T06:59:51.8422423Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:59:51.8445362Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:59:51.8449252Z ##[group]Initializing the repository 2024-06-26T06:59:51.8452124Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:59:51.8472747Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T06:59:51.8474268Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T06:59:51.8475513Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T06:59:51.8476353Z hint: 2024-06-26T06:59:51.8477155Z hint: git config --global init.defaultBranch 2024-06-26T06:59:51.8477711Z hint: 2024-06-26T06:59:51.8478308Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T06:59:51.8479297Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T06:59:51.8480398Z hint: 2024-06-26T06:59:51.8480815Z hint: git branch -m 2024-06-26T06:59:51.8481651Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T06:59:51.8483563Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T06:59:51.8503518Z ##[endgroup] 2024-06-26T06:59:51.8504237Z ##[group]Disabling automatic garbage collection 2024-06-26T06:59:51.8506068Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T06:59:51.8525554Z ##[endgroup] 2024-06-26T06:59:51.8526214Z ##[group]Setting up auth 2024-06-26T06:59:51.8531619Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T06:59:51.8554064Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T06:59:51.8760284Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T06:59:51.8783209Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T06:59:51.8985697Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T06:59:51.9037618Z ##[endgroup] 2024-06-26T06:59:51.9038351Z ##[group]Fetching the repository 2024-06-26T06:59:51.9043469Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T06:59:53.9842324Z remote: Enumerating objects: 987392 2024-06-26T06:59:53.9843048Z remote: Enumerating objects: 990209, done. 2024-06-26T06:59:53.9844101Z remote: Counting objects: 0% (1/2817) 2024-06-26T06:59:53.9844958Z remote: Counting objects: 1% (29/2817) 2024-06-26T06:59:53.9845837Z remote: Counting objects: 2% (57/2817) 2024-06-26T06:59:53.9846690Z remote: Counting objects: 3% (85/2817) 2024-06-26T06:59:53.9847531Z remote: Counting objects: 4% (113/2817) 2024-06-26T06:59:53.9848143Z remote: Counting objects: 5% (141/2817) 2024-06-26T06:59:53.9848744Z remote: Counting objects: 6% (170/2817) 2024-06-26T06:59:53.9849355Z remote: Counting objects: 7% (198/2817) 2024-06-26T06:59:53.9849974Z remote: Counting objects: 8% (226/2817) 2024-06-26T06:59:53.9850830Z remote: Counting objects: 9% (254/2817) 2024-06-26T06:59:53.9851643Z remote: Counting objects: 10% (282/2817) 2024-06-26T06:59:53.9852460Z remote: Counting objects: 11% (310/2817) 2024-06-26T06:59:53.9853251Z remote: Counting objects: 12% (339/2817) 2024-06-26T06:59:53.9854052Z remote: Counting objects: 13% (367/2817) 2024-06-26T06:59:53.9854868Z remote: Counting objects: 14% (395/2817) 2024-06-26T06:59:53.9855670Z remote: Counting objects: 15% (423/2817) 2024-06-26T06:59:53.9856460Z remote: Counting objects: 16% (451/2817) 2024-06-26T06:59:53.9857236Z remote: Counting objects: 17% (479/2817) 2024-06-26T06:59:53.9858060Z remote: Counting objects: 18% (508/2817) 2024-06-26T06:59:53.9858848Z remote: Counting objects: 19% (536/2817) 2024-06-26T06:59:53.9859443Z remote: Counting objects: 20% (564/2817) 2024-06-26T06:59:53.9860053Z remote: Counting objects: 21% (592/2817) 2024-06-26T06:59:53.9860646Z remote: Counting objects: 22% (620/2817) 2024-06-26T06:59:53.9861235Z remote: Counting objects: 23% (648/2817) 2024-06-26T06:59:53.9861819Z remote: Counting objects: 24% (677/2817) 2024-06-26T06:59:53.9862413Z remote: Counting objects: 25% (705/2817) 2024-06-26T06:59:53.9863000Z remote: Counting objects: 26% (733/2817) 2024-06-26T06:59:53.9863592Z remote: Counting objects: 27% (761/2817) 2024-06-26T06:59:53.9864459Z remote: Counting objects: 28% (789/2817) 2024-06-26T06:59:53.9865119Z remote: Counting objects: 29% (817/2817) 2024-06-26T06:59:53.9865706Z remote: Counting objects: 30% (846/2817) 2024-06-26T06:59:53.9866300Z remote: Counting objects: 31% (874/2817) 2024-06-26T06:59:53.9866881Z remote: Counting objects: 32% (902/2817) 2024-06-26T06:59:53.9867477Z remote: Counting objects: 33% (930/2817) 2024-06-26T06:59:53.9868066Z remote: Counting objects: 34% (958/2817) 2024-06-26T06:59:53.9868724Z remote: Counting objects: 35% (986/2817) 2024-06-26T06:59:53.9869519Z remote: Counting objects: 36% (1015/2817) 2024-06-26T06:59:53.9870334Z remote: Counting objects: 37% (1043/2817) 2024-06-26T06:59:53.9871166Z remote: Counting objects: 38% (1071/2817) 2024-06-26T06:59:53.9871978Z remote: Counting objects: 39% (1099/2817) 2024-06-26T06:59:53.9872797Z remote: Counting objects: 40% (1127/2817) 2024-06-26T06:59:53.9873615Z remote: Counting objects: 41% (1155/2817) 2024-06-26T06:59:53.9874722Z remote: Counting objects: 42% (1184/2817) 2024-06-26T06:59:53.9875504Z remote: Counting objects: 43% (1212/2817) 2024-06-26T06:59:53.9876260Z remote: Counting objects: 44% (1240/2817) 2024-06-26T06:59:53.9877048Z remote: Counting objects: 45% (1268/2817) 2024-06-26T06:59:53.9877892Z remote: Counting objects: 46% (1296/2817) 2024-06-26T06:59:53.9878718Z remote: Counting objects: 47% (1324/2817) 2024-06-26T06:59:53.9879342Z remote: Counting objects: 48% (1353/2817) 2024-06-26T06:59:53.9880216Z remote: Counting objects: 49% (1381/2817) 2024-06-26T06:59:53.9880836Z remote: Counting objects: 50% (1409/2817) 2024-06-26T06:59:53.9881437Z remote: Counting objects: 51% (1437/2817) 2024-06-26T06:59:53.9882035Z remote: Counting objects: 52% (1465/2817) 2024-06-26T06:59:53.9882643Z remote: Counting objects: 53% (1494/2817) 2024-06-26T06:59:53.9883247Z remote: Counting objects: 54% (1522/2817) 2024-06-26T06:59:53.9883850Z remote: Counting objects: 55% (1550/2817) 2024-06-26T06:59:53.9884627Z remote: Counting objects: 56% (1578/2817) 2024-06-26T06:59:53.9885240Z remote: Counting objects: 57% (1606/2817) 2024-06-26T06:59:53.9885851Z remote: Counting objects: 58% (1634/2817) 2024-06-26T06:59:53.9886469Z remote: Counting objects: 59% (1663/2817) 2024-06-26T06:59:53.9887071Z remote: Counting objects: 60% (1691/2817) 2024-06-26T06:59:53.9887681Z remote: Counting objects: 61% (1719/2817) 2024-06-26T06:59:53.9888294Z remote: Counting objects: 62% (1747/2817) 2024-06-26T06:59:53.9888918Z remote: Counting objects: 63% (1775/2817) 2024-06-26T06:59:53.9889713Z remote: Counting objects: 64% (1803/2817) 2024-06-26T06:59:53.9890402Z remote: Counting objects: 65% (1832/2817) 2024-06-26T06:59:53.9891021Z remote: Counting objects: 66% (1860/2817) 2024-06-26T06:59:53.9891655Z remote: Counting objects: 67% (1888/2817) 2024-06-26T06:59:53.9892332Z remote: Counting objects: 68% (1916/2817) 2024-06-26T06:59:53.9893177Z remote: Counting objects: 69% (1944/2817) 2024-06-26T06:59:53.9893891Z remote: Counting objects: 70% (1972/2817) 2024-06-26T06:59:53.9894530Z remote: Counting objects: 71% (2001/2817) 2024-06-26T06:59:53.9895343Z remote: Counting objects: 72% (2029/2817) 2024-06-26T06:59:53.9896146Z remote: Counting objects: 73% (2057/2817) 2024-06-26T06:59:53.9896852Z remote: Counting objects: 74% (2085/2817) 2024-06-26T06:59:53.9897455Z remote: Counting objects: 75% (2113/2817) 2024-06-26T06:59:53.9898059Z remote: Counting objects: 76% (2141/2817) 2024-06-26T06:59:53.9898655Z remote: Counting objects: 77% (2170/2817) 2024-06-26T06:59:53.9899255Z remote: Counting objects: 78% (2198/2817) 2024-06-26T06:59:53.9899987Z remote: Counting objects: 79% (2226/2817) 2024-06-26T06:59:53.9900599Z remote: Counting objects: 80% (2254/2817) 2024-06-26T06:59:53.9901198Z remote: Counting objects: 81% (2282/2817) 2024-06-26T06:59:53.9901799Z remote: Counting objects: 82% (2310/2817) 2024-06-26T06:59:53.9902402Z remote: Counting objects: 83% (2339/2817) 2024-06-26T06:59:53.9903018Z remote: Counting objects: 84% (2367/2817) 2024-06-26T06:59:53.9903619Z remote: Counting objects: 85% (2395/2817) 2024-06-26T06:59:53.9904224Z remote: Counting objects: 86% (2423/2817) 2024-06-26T06:59:53.9904830Z remote: Counting objects: 87% (2451/2817) 2024-06-26T06:59:53.9905431Z remote: Counting objects: 88% (2479/2817) 2024-06-26T06:59:53.9906029Z remote: Counting objects: 89% (2508/2817) 2024-06-26T06:59:53.9906639Z remote: Counting objects: 90% (2536/2817) 2024-06-26T06:59:53.9907251Z remote: Counting objects: 91% (2564/2817) 2024-06-26T06:59:53.9907859Z remote: Counting objects: 92% (2592/2817) 2024-06-26T06:59:53.9908461Z remote: Counting objects: 93% (2620/2817) 2024-06-26T06:59:53.9909073Z remote: Counting objects: 94% (2648/2817) 2024-06-26T06:59:53.9909695Z remote: Counting objects: 95% (2677/2817) 2024-06-26T06:59:53.9910328Z remote: Counting objects: 96% (2705/2817) 2024-06-26T06:59:53.9910995Z remote: Counting objects: 97% (2733/2817) 2024-06-26T06:59:53.9911598Z remote: Counting objects: 98% (2761/2817) 2024-06-26T06:59:53.9912197Z remote: Counting objects: 99% (2789/2817) 2024-06-26T06:59:53.9912799Z remote: Counting objects: 100% (2817/2817) 2024-06-26T06:59:53.9913431Z remote: Counting objects: 100% (2817/2817), done. 2024-06-26T06:59:54.0065971Z remote: Compressing objects: 0% (1/1140) 2024-06-26T06:59:54.0649015Z remote: Compressing objects: 1% (12/1140) 2024-06-26T06:59:54.1061134Z remote: Compressing objects: 2% (23/1140) 2024-06-26T06:59:54.1380584Z remote: Compressing objects: 3% (35/1140) 2024-06-26T06:59:54.1563608Z remote: Compressing objects: 4% (46/1140) 2024-06-26T06:59:54.1850687Z remote: Compressing objects: 5% (57/1140) 2024-06-26T06:59:54.2069238Z remote: Compressing objects: 6% (69/1140) 2024-06-26T06:59:54.3096035Z remote: Compressing objects: 7% (80/1140) 2024-06-26T06:59:54.3814000Z remote: Compressing objects: 8% (92/1140) 2024-06-26T06:59:54.4435795Z remote: Compressing objects: 9% (103/1140) 2024-06-26T06:59:54.4672808Z remote: Compressing objects: 10% (114/1140) 2024-06-26T06:59:54.4799144Z remote: Compressing objects: 11% (126/1140) 2024-06-26T06:59:54.4892111Z remote: Compressing objects: 12% (137/1140) 2024-06-26T06:59:54.4950236Z remote: Compressing objects: 13% (149/1140) 2024-06-26T06:59:54.4951823Z remote: Compressing objects: 14% (160/1140) 2024-06-26T06:59:54.5034126Z remote: Compressing objects: 15% (171/1140) 2024-06-26T06:59:54.5035607Z remote: Compressing objects: 16% (183/1140) 2024-06-26T06:59:54.5039288Z remote: Compressing objects: 17% (194/1140) 2024-06-26T06:59:54.5072315Z remote: Compressing objects: 18% (206/1140) 2024-06-26T06:59:54.5096904Z remote: Compressing objects: 19% (217/1140) 2024-06-26T06:59:54.5121981Z remote: Compressing objects: 20% (228/1140) 2024-06-26T06:59:54.5141302Z remote: Compressing objects: 21% (240/1140) 2024-06-26T06:59:54.5161652Z remote: Compressing objects: 22% (251/1140) 2024-06-26T06:59:54.5171309Z remote: Compressing objects: 23% (263/1140) 2024-06-26T06:59:54.5178698Z remote: Compressing objects: 24% (274/1140) 2024-06-26T06:59:54.5193563Z remote: Compressing objects: 25% (285/1140) 2024-06-26T06:59:54.5196920Z remote: Compressing objects: 26% (297/1140) 2024-06-26T06:59:54.5201827Z remote: Compressing objects: 27% (308/1140) 2024-06-26T06:59:54.5208469Z remote: Compressing objects: 28% (320/1140) 2024-06-26T06:59:54.5217385Z remote: Compressing objects: 29% (331/1140) 2024-06-26T06:59:54.5227005Z remote: Compressing objects: 30% (342/1140) 2024-06-26T06:59:54.5229080Z remote: Compressing objects: 31% (354/1140) 2024-06-26T06:59:54.5245255Z remote: Compressing objects: 32% (365/1140) 2024-06-26T06:59:54.5257876Z remote: Compressing objects: 33% (377/1140) 2024-06-26T06:59:54.5272519Z remote: Compressing objects: 34% (388/1140) 2024-06-26T06:59:54.5287060Z remote: Compressing objects: 35% (399/1140) 2024-06-26T06:59:54.5309160Z remote: Compressing objects: 36% (411/1140) 2024-06-26T06:59:54.5330953Z remote: Compressing objects: 37% (422/1140) 2024-06-26T06:59:54.5348483Z remote: Compressing objects: 38% (434/1140) 2024-06-26T06:59:54.5355887Z remote: Compressing objects: 39% (445/1140) 2024-06-26T06:59:54.5375815Z remote: Compressing objects: 40% (456/1140) 2024-06-26T06:59:54.5402783Z remote: Compressing objects: 41% (468/1140) 2024-06-26T06:59:54.5418289Z remote: Compressing objects: 42% (479/1140) 2024-06-26T06:59:54.5431485Z remote: Compressing objects: 43% (491/1140) 2024-06-26T06:59:54.5434782Z remote: Compressing objects: 44% (502/1140) 2024-06-26T06:59:54.5440504Z remote: Compressing objects: 45% (513/1140) 2024-06-26T06:59:54.5452453Z remote: Compressing objects: 46% (525/1140) 2024-06-26T06:59:54.5466531Z remote: Compressing objects: 47% (536/1140) 2024-06-26T06:59:54.5471830Z remote: Compressing objects: 48% (548/1140) 2024-06-26T06:59:54.5475620Z remote: Compressing objects: 49% (559/1140) 2024-06-26T06:59:54.5483611Z remote: Compressing objects: 50% (570/1140) 2024-06-26T06:59:54.5494924Z remote: Compressing objects: 51% (582/1140) 2024-06-26T06:59:54.5515144Z remote: Compressing objects: 52% (593/1140) 2024-06-26T06:59:54.5529500Z remote: Compressing objects: 53% (605/1140) 2024-06-26T06:59:54.5547823Z remote: Compressing objects: 54% (616/1140) 2024-06-26T06:59:54.5560238Z remote: Compressing objects: 55% (627/1140) 2024-06-26T06:59:54.5566478Z remote: Compressing objects: 56% (639/1140) 2024-06-26T06:59:54.5576861Z remote: Compressing objects: 57% (650/1140) 2024-06-26T06:59:54.5588335Z remote: Compressing objects: 58% (662/1140) 2024-06-26T06:59:54.5599693Z remote: Compressing objects: 59% (673/1140) 2024-06-26T06:59:54.5610562Z remote: Compressing objects: 60% (684/1140) 2024-06-26T06:59:54.5619729Z remote: Compressing objects: 61% (696/1140) 2024-06-26T06:59:54.5632545Z remote: Compressing objects: 62% (707/1140) 2024-06-26T06:59:54.5643381Z remote: Compressing objects: 63% (719/1140) 2024-06-26T06:59:54.5652205Z remote: Compressing objects: 64% (730/1140) 2024-06-26T06:59:54.5666305Z remote: Compressing objects: 65% (741/1140) 2024-06-26T06:59:54.5675061Z remote: Compressing objects: 66% (753/1140) 2024-06-26T06:59:54.5677405Z remote: Compressing objects: 67% (764/1140) 2024-06-26T06:59:54.5678657Z remote: Compressing objects: 68% (776/1140) 2024-06-26T06:59:54.5688382Z remote: Compressing objects: 69% (787/1140) 2024-06-26T06:59:54.5692547Z remote: Compressing objects: 70% (798/1140) 2024-06-26T06:59:54.5695254Z remote: Compressing objects: 71% (810/1140) 2024-06-26T06:59:54.5699961Z remote: Compressing objects: 72% (821/1140) 2024-06-26T06:59:54.5702093Z remote: Compressing objects: 73% (833/1140) 2024-06-26T06:59:54.5703580Z remote: Compressing objects: 74% (844/1140) 2024-06-26T06:59:54.5705354Z remote: Compressing objects: 75% (855/1140) 2024-06-26T06:59:54.5706094Z remote: Compressing objects: 76% (867/1140) 2024-06-26T06:59:54.5706845Z remote: Compressing objects: 77% (878/1140) 2024-06-26T06:59:54.5707478Z remote: Compressing objects: 78% (890/1140) 2024-06-26T06:59:54.5708107Z remote: Compressing objects: 79% (901/1140) 2024-06-26T06:59:54.5710761Z remote: Compressing objects: 80% (912/1140) 2024-06-26T06:59:54.5711415Z remote: Compressing objects: 81% (924/1140) 2024-06-26T06:59:54.5715061Z remote: Compressing objects: 82% (935/1140) 2024-06-26T06:59:54.5717781Z remote: Compressing objects: 83% (947/1140) 2024-06-26T06:59:54.5718580Z remote: Compressing objects: 84% (958/1140) 2024-06-26T06:59:54.5720143Z remote: Compressing objects: 85% (969/1140) 2024-06-26T06:59:54.5723492Z remote: Compressing objects: 86% (981/1140) 2024-06-26T06:59:54.5724800Z remote: Compressing objects: 87% (992/1140) 2024-06-26T06:59:54.5729655Z remote: Compressing objects: 88% (1004/1140) 2024-06-26T06:59:54.5730359Z remote: Compressing objects: 89% (1015/1140) 2024-06-26T06:59:54.5732339Z remote: Compressing objects: 90% (1026/1140) 2024-06-26T06:59:54.5733037Z remote: Compressing objects: 91% (1038/1140) 2024-06-26T06:59:54.5733814Z remote: Compressing objects: 92% (1049/1140) 2024-06-26T06:59:54.5734531Z remote: Compressing objects: 93% (1061/1140) 2024-06-26T06:59:54.5736433Z remote: Compressing objects: 94% (1072/1140) 2024-06-26T06:59:54.5737214Z remote: Compressing objects: 95% (1083/1140) 2024-06-26T06:59:54.5738317Z remote: Compressing objects: 96% (1095/1140) 2024-06-26T06:59:54.5738998Z remote: Compressing objects: 97% (1106/1140) 2024-06-26T06:59:54.5741132Z remote: Compressing objects: 98% (1118/1140) 2024-06-26T06:59:54.5741826Z remote: Compressing objects: 99% (1129/1140) 2024-06-26T06:59:54.5742456Z remote: Compressing objects: 100% (1140/1140) 2024-06-26T06:59:54.5743134Z remote: Compressing objects: 100% (1140/1140), done. 2024-06-26T07:00:14.1572065Z remote: Total 990209 (delta 2280), reused 2016 (delta 1676), pack-reused 987392 2024-06-26T07:00:39.8962133Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:00:39.8984125Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:39.8987508Z ##[endgroup] 2024-06-26T07:00:39.8988064Z ##[group]Determining the checkout info 2024-06-26T07:00:39.8990532Z ##[endgroup] 2024-06-26T07:00:39.8991080Z ##[group]Checking out the ref 2024-06-26T07:00:39.8992511Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:41.1267455Z ##[endgroup] 2024-06-26T07:00:41.1268913Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:00:41.1272557Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:00:41.1315794Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:00:41.1336777Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:00:41.1355997Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:00:41.1372730Z ##[endgroup] 2024-06-26T07:00:41.1373255Z ##[group]Fetching submodules 2024-06-26T07:00:41.1376393Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:00:41.1588384Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:00:41.1790502Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:00:41.1793122Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:00:41.1795166Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:00:41.1796896Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:00:41.1798593Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:00:41.1800232Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:00:41.1801596Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:00:41.1803010Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:00:41.1804576Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:00:41.1806343Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:00:41.1808159Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:00:41.1810667Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:00:41.1813014Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:00:41.1815643Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:00:41.1817935Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:00:41.1820547Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:00:41.1823372Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:00:41.1825984Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:00:41.1828804Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:00:41.1831520Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:00:41.1834680Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:00:41.1837953Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:00:41.1841089Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:00:41.1844122Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:00:41.1847533Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:00:41.1850681Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:00:41.1854235Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:00:41.1857440Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:00:41.1861175Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:00:41.1864656Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:00:41.1868471Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:00:41.1871889Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:00:41.1876125Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:00:41.1879639Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:00:41.1883592Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:00:41.1903079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:00:41.4367411Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:00:41.6722589Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:00:41.8583011Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:00:42.0780235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:00:43.9689411Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:00:53.7712435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:00:54.1502725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:00:54.6231963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:00:55.1739615Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:00:56.2762007Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:00:57.9324025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:01:03.7456086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:01:05.0228080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:01:06.7623591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:01:07.9240227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:01:08.0756499Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:01:08.4690343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:01:08.7838243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:01:09.7458774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:01:10.0458171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:01:10.2897157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:01:11.9477768Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:01:12.6471435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:01:13.3943571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:01:19.1771923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:01:21.4733892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:01:27.0444454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:01:27.2387497Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:01:35.3710906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:01:35.5533554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:01:35.8113146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:01:36.7808644Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:01:37.0514406Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:01:37.6322451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:01:38.0305615Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:01:38.0388000Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:01:38.0447817Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:01:38.0628718Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:01:38.0935115Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:01:38.8757380Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:01:38.8930313Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:01:38.9278140Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:01:39.0128839Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:01:39.0398780Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:01:39.4693577Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:01:39.6829599Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:01:39.7455077Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:01:39.7466813Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:01:39.7470354Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:01:39.7471873Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:01:39.7473408Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:01:39.7475426Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:01:39.7490595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:01:40.6430494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:01:41.2247664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:01:42.9215268Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:01:43.9018228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:01:44.2189357Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:01:44.3023816Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:01:44.6473904Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:01:44.7016804Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:01:44.7100883Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:01:44.8011303Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:01:44.8293359Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:01:44.8358038Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:01:44.8663696Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:01:44.8857726Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:01:44.9217466Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:01:44.9310206Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:01:44.9321566Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:01:44.9335660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:01:56.6725584Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:01:56.6857912Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:01:56.7638481Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:01:56.7648795Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:01:56.7650619Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:01:56.7652612Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:01:56.7668095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:01:57.3009652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:01:58.4974311Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:01:59.5554299Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:01:59.5563491Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:01:59.5565401Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:01:59.5567182Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:01:59.5569311Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:01:59.5571358Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:01:59.5573674Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:01:59.5575783Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:01:59.5577777Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:01:59.5596135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:02:00.5743394Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:02:00.9140349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:02:02.1096669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:02:02.3896602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:02:02.9273705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:02:03.8779727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:02:09.4580149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:02:09.8206273Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:02:09.8345602Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:02:09.8635523Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:02:09.8733284Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:02:09.8743864Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:09.8760206Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:02:10.1635717Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:02:10.1774272Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:02:10.2106967Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:02:10.2918206Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:02:10.3038547Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:02:10.3343073Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:02:10.3840236Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:02:10.4152848Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:02:10.4347770Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:02:10.5213516Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:02:10.7924728Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:02:10.7950476Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:10.7952090Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:10.7969628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:02:11.1707776Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:02:12.2504662Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:02:12.2784330Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:02:12.3312779Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:02:12.3324235Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:12.3330569Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:12.3332730Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:12.3334614Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:12.3336796Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:12.3339075Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:12.3340998Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:12.3342623Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:12.3354982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:02:12.8099648Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:02:14.0789911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:02:14.4422869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:02:19.9869653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:02:20.2662743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:02:20.4443925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:02:20.7673340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:02:28.2019146Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:02:28.2355700Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:02:28.2483072Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:02:28.3321116Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:02:28.3419342Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:02:28.3524907Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:02:28.3637654Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:02:28.3647367Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:28.3649338Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:28.3665401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:02:30.2028124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:02:31.3916254Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:02:31.4282704Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:02:31.8001374Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:02:31.8088925Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:02:32.0265797Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:02:32.0278325Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:32.0280276Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:32.0295668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:02:32.4153442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:02:33.4034418Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:02:33.4656604Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:02:33.4715698Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:02:33.4803697Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:02:33.5081705Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:02:33.5304063Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:02:33.5632343Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:02:33.5833082Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:02:33.5844671Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:33.5846421Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:33.5848675Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:33.5851005Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:33.5867883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:02:34.5349581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:02:34.7332464Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:02:35.7926041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:02:36.9144769Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:02:36.9257077Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:02:36.9823920Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:02:37.0056743Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:02:37.0067643Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:37.0084981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:02:37.1791696Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:02:37.1814848Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:02:37.2023374Z Entering 'android/libs/fbjni' 2024-06-26T07:02:37.2051959Z Entering 'third_party/FP16' 2024-06-26T07:02:37.2078100Z Entering 'third_party/FXdiv' 2024-06-26T07:02:37.2104302Z Entering 'third_party/NNPACK' 2024-06-26T07:02:37.2131746Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:37.2157291Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:37.2198042Z Entering 'third_party/benchmark' 2024-06-26T07:02:37.2224639Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:37.2253912Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:37.2280711Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:37.2310098Z Entering 'third_party/cutlass' 2024-06-26T07:02:37.2342816Z Entering 'third_party/eigen' 2024-06-26T07:02:37.2372226Z Entering 'third_party/fbgemm' 2024-06-26T07:02:37.2400685Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:37.2427614Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:37.2454647Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:37.2485666Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:37.2512566Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:37.2539633Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:37.2570410Z Entering 'third_party/fmt' 2024-06-26T07:02:37.2598552Z Entering 'third_party/foxi' 2024-06-26T07:02:37.2624838Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:37.2653111Z Entering 'third_party/gloo' 2024-06-26T07:02:37.2680597Z Entering 'third_party/googletest' 2024-06-26T07:02:37.2708312Z Entering 'third_party/ideep' 2024-06-26T07:02:37.2735736Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:37.2767163Z Entering 'third_party/ittapi' 2024-06-26T07:02:37.2796963Z Entering 'third_party/kineto' 2024-06-26T07:02:37.2823797Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:37.2850561Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:37.2879325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:37.2905462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:37.2932671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:37.2957718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:37.2986643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:37.3013719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:37.3042584Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:37.3070988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:37.3098751Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:37.3126324Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:37.3152671Z Entering 'third_party/mimalloc' 2024-06-26T07:02:37.3182302Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:37.3209504Z Entering 'third_party/nlohmann' 2024-06-26T07:02:37.3238857Z Entering 'third_party/onnx' 2024-06-26T07:02:37.3280096Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:37.3311538Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:37.3343387Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:37.3372092Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:37.3399657Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:37.3426612Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:37.3453066Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:37.3480256Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:37.3507599Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:37.3534243Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:37.3561239Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:37.3588908Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:37.3618792Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:37.3660915Z Entering 'third_party/pocketfft' 2024-06-26T07:02:37.3689345Z Entering 'third_party/protobuf' 2024-06-26T07:02:37.3719828Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:37.3746640Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:37.3775810Z Entering 'third_party/psimd' 2024-06-26T07:02:37.3803499Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:37.3830596Z Entering 'third_party/pybind11' 2024-06-26T07:02:37.3857478Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:37.3885791Z Entering 'third_party/sleef' 2024-06-26T07:02:37.3915398Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:37.3943290Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:37.3969537Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:37.4001142Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:37.4028779Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:37.4056332Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:37.4098368Z ##[endgroup] 2024-06-26T07:02:37.4100390Z ##[group]Persisting credentials for submodules 2024-06-26T07:02:37.4102574Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:02:37.4317774Z Entering 'android/libs/fbjni' 2024-06-26T07:02:37.4352574Z Entering 'third_party/FP16' 2024-06-26T07:02:37.4386812Z Entering 'third_party/FXdiv' 2024-06-26T07:02:37.4421135Z Entering 'third_party/NNPACK' 2024-06-26T07:02:37.4456749Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:37.4493278Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:37.4538785Z Entering 'third_party/benchmark' 2024-06-26T07:02:37.4573532Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:37.4605941Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:37.4641283Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:37.4672793Z Entering 'third_party/cutlass' 2024-06-26T07:02:37.4712697Z Entering 'third_party/eigen' 2024-06-26T07:02:37.4747376Z Entering 'third_party/fbgemm' 2024-06-26T07:02:37.4782163Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:37.4816947Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:37.4851190Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:37.4889082Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:37.4922805Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:37.4957399Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:37.4992424Z Entering 'third_party/fmt' 2024-06-26T07:02:37.5027132Z Entering 'third_party/foxi' 2024-06-26T07:02:37.5063787Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:37.5100025Z Entering 'third_party/gloo' 2024-06-26T07:02:37.5134422Z Entering 'third_party/googletest' 2024-06-26T07:02:37.5168604Z Entering 'third_party/ideep' 2024-06-26T07:02:37.5202443Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:37.5242736Z Entering 'third_party/ittapi' 2024-06-26T07:02:37.5277714Z Entering 'third_party/kineto' 2024-06-26T07:02:37.5310662Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:37.5345272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:37.5381008Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:37.5416317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:37.5450894Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:37.5483731Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:37.5519032Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:37.5553218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:37.5587942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:37.5622948Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:37.5656204Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:37.5688047Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:37.5722271Z Entering 'third_party/mimalloc' 2024-06-26T07:02:37.5757759Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:37.5790212Z Entering 'third_party/nlohmann' 2024-06-26T07:02:37.5823660Z Entering 'third_party/onnx' 2024-06-26T07:02:37.5872436Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:37.5906992Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:37.5943807Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:37.5980748Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:37.6015451Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:37.6048889Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:37.6083310Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:37.6116134Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:37.6149111Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:37.6182474Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:37.6216477Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:37.6250806Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:37.6286866Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:37.6335793Z Entering 'third_party/pocketfft' 2024-06-26T07:02:37.6372285Z Entering 'third_party/protobuf' 2024-06-26T07:02:37.6406643Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:37.6440012Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:37.6473426Z Entering 'third_party/psimd' 2024-06-26T07:02:37.6508340Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:37.6543204Z Entering 'third_party/pybind11' 2024-06-26T07:02:37.6576196Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:37.6611190Z Entering 'third_party/sleef' 2024-06-26T07:02:37.6645407Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:37.6681949Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:37.6715740Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:37.6749619Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:37.6785764Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:37.6818802Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:37.6867509Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:02:37.7070564Z Entering 'android/libs/fbjni' 2024-06-26T07:02:37.7101050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:02:37.7114594Z Entering 'third_party/FP16' 2024-06-26T07:02:37.7144791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:02:37.7159190Z Entering 'third_party/FXdiv' 2024-06-26T07:02:37.7190248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:02:37.7202524Z Entering 'third_party/NNPACK' 2024-06-26T07:02:37.7232096Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:02:37.7245283Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:37.7276729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:02:37.7289306Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:37.7319307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:02:37.7342379Z Entering 'third_party/benchmark' 2024-06-26T07:02:37.7374262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:37.7387469Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:37.7420476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:02:37.7433711Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:37.7466260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:02:37.7479200Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:37.7512201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:02:37.7525387Z Entering 'third_party/cutlass' 2024-06-26T07:02:37.7556670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:02:37.7574716Z Entering 'third_party/eigen' 2024-06-26T07:02:37.7606404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:02:37.7621055Z Entering 'third_party/fbgemm' 2024-06-26T07:02:37.7652968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:02:37.7665660Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:37.7696897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:02:37.7709891Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:37.7741839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:02:37.7754491Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:37.7784696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:02:37.7803440Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:37.7834129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:37.7847014Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:37.7876704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:02:37.7889628Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:37.7922211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:02:37.7937394Z Entering 'third_party/fmt' 2024-06-26T07:02:37.7967984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:02:37.7981805Z Entering 'third_party/foxi' 2024-06-26T07:02:37.8014113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:02:37.8025588Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:37.8056248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:02:37.8068806Z Entering 'third_party/gloo' 2024-06-26T07:02:37.8101393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:02:37.8115039Z Entering 'third_party/googletest' 2024-06-26T07:02:37.8146844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:37.8159048Z Entering 'third_party/ideep' 2024-06-26T07:02:37.8190175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:02:37.8201513Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:37.8230698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:02:37.8247980Z Entering 'third_party/ittapi' 2024-06-26T07:02:37.8279831Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:02:37.8292866Z Entering 'third_party/kineto' 2024-06-26T07:02:37.8331455Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:02:37.8335521Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:37.8367338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:02:37.8379949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:37.8411958Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:02:37.8425020Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:37.8457882Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:02:37.8469322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:37.8501859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:02:37.8514936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:37.8547425Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:02:37.8559246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:37.8590454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:02:37.8603767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:37.8635275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:02:37.8648264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:37.8679269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:37.8692699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:37.8724042Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:02:37.8737822Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:37.8771106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:02:37.8784718Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:37.8816316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:02:37.8828299Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:37.8860559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:02:37.8875181Z Entering 'third_party/mimalloc' 2024-06-26T07:02:37.8906752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:02:37.8919781Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:37.8950585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:02:37.8964682Z Entering 'third_party/nlohmann' 2024-06-26T07:02:37.8993644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:02:37.9006934Z Entering 'third_party/onnx' 2024-06-26T07:02:37.9039659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:02:37.9064850Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:37.9099300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:37.9111935Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:37.9143926Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:37.9159132Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:37.9190570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:02:37.9203126Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:37.9234987Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:37.9247435Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:37.9278169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:37.9290450Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:37.9322116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:02:37.9334072Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:37.9365072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:02:37.9378745Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:37.9409970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:02:37.9422728Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:37.9454643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:02:37.9467313Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:37.9499982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:02:37.9511456Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:37.9544706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:02:37.9559316Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:37.9590351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:02:37.9602760Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:37.9635030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:02:37.9662372Z Entering 'third_party/pocketfft' 2024-06-26T07:02:37.9693205Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:02:37.9707524Z Entering 'third_party/protobuf' 2024-06-26T07:02:37.9738485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:02:37.9751917Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:37.9783030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:37.9796238Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:37.9827176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:37.9841141Z Entering 'third_party/psimd' 2024-06-26T07:02:37.9874428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:02:37.9888516Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:37.9920302Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:02:37.9933803Z Entering 'third_party/pybind11' 2024-06-26T07:02:37.9965002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:37.9978859Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:38.0010008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:02:38.0022324Z Entering 'third_party/sleef' 2024-06-26T07:02:38.0053310Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:02:38.0066679Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:38.0098404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:02:38.0110395Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:38.0140214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:38.0153107Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:38.0185683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:02:38.0198185Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:38.0229527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:02:38.0242462Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:38.0275210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:38.0287347Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:38.0319534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:02:38.1208231Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:02:38.1419607Z Entering 'android/libs/fbjni' 2024-06-26T07:02:38.1446006Z Entering 'third_party/FP16' 2024-06-26T07:02:38.1473605Z Entering 'third_party/FXdiv' 2024-06-26T07:02:38.1500230Z Entering 'third_party/NNPACK' 2024-06-26T07:02:38.1525107Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:38.1550554Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:38.1588820Z Entering 'third_party/benchmark' 2024-06-26T07:02:38.1615665Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:38.1642658Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:38.1667647Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:38.1694983Z Entering 'third_party/cutlass' 2024-06-26T07:02:38.1728119Z Entering 'third_party/eigen' 2024-06-26T07:02:38.1757888Z Entering 'third_party/fbgemm' 2024-06-26T07:02:38.1785610Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:38.1812261Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:38.1840749Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:38.1872908Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:38.1900629Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:38.1929800Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:38.1959090Z Entering 'third_party/fmt' 2024-06-26T07:02:38.1987589Z Entering 'third_party/foxi' 2024-06-26T07:02:38.2014218Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:38.2042581Z Entering 'third_party/gloo' 2024-06-26T07:02:38.2070975Z Entering 'third_party/googletest' 2024-06-26T07:02:38.2098930Z Entering 'third_party/ideep' 2024-06-26T07:02:38.2126730Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:38.2158470Z Entering 'third_party/ittapi' 2024-06-26T07:02:38.2187142Z Entering 'third_party/kineto' 2024-06-26T07:02:38.2214195Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:38.2241445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:38.2270759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:38.2299974Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:38.2326946Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:38.2352694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:38.2382215Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:38.2408766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:38.2437558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:38.2465784Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:38.2494497Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:38.2522614Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:38.2552421Z Entering 'third_party/mimalloc' 2024-06-26T07:02:38.2582531Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:38.2609729Z Entering 'third_party/nlohmann' 2024-06-26T07:02:38.2639294Z Entering 'third_party/onnx' 2024-06-26T07:02:38.2678208Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:38.2706277Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:38.2735545Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:38.2765135Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:38.2791462Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:38.2819774Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:38.2846371Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:38.2872912Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:38.2898283Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:38.2924608Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:38.2950267Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:38.2979534Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:38.3007120Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:38.3048762Z Entering 'third_party/pocketfft' 2024-06-26T07:02:38.3076245Z Entering 'third_party/protobuf' 2024-06-26T07:02:38.3104625Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:38.3132646Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:38.3162051Z Entering 'third_party/psimd' 2024-06-26T07:02:38.3189880Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:38.3218929Z Entering 'third_party/pybind11' 2024-06-26T07:02:38.3247144Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:38.3274383Z Entering 'third_party/sleef' 2024-06-26T07:02:38.3302485Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:38.3330986Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:38.3358089Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:38.3384589Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:38.3411044Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:38.3436343Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:38.3478068Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:02:38.3683291Z Entering 'android/libs/fbjni' 2024-06-26T07:02:38.3713088Z Entering 'third_party/FP16' 2024-06-26T07:02:38.3740843Z Entering 'third_party/FXdiv' 2024-06-26T07:02:38.3767983Z Entering 'third_party/NNPACK' 2024-06-26T07:02:38.3795479Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:38.3824630Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:38.3862222Z Entering 'third_party/benchmark' 2024-06-26T07:02:38.3890211Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:38.3918527Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:38.3946161Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:38.3973110Z Entering 'third_party/cutlass' 2024-06-26T07:02:38.4007161Z Entering 'third_party/eigen' 2024-06-26T07:02:38.4035512Z Entering 'third_party/fbgemm' 2024-06-26T07:02:38.4063879Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:38.4091407Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:38.4119711Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:38.4150511Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:38.4178128Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:38.4205403Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:38.4235044Z Entering 'third_party/fmt' 2024-06-26T07:02:38.4263622Z Entering 'third_party/foxi' 2024-06-26T07:02:38.4291105Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:38.4319204Z Entering 'third_party/gloo' 2024-06-26T07:02:38.4348068Z Entering 'third_party/googletest' 2024-06-26T07:02:38.4376097Z Entering 'third_party/ideep' 2024-06-26T07:02:38.4401418Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:38.4435234Z Entering 'third_party/ittapi' 2024-06-26T07:02:38.4462196Z Entering 'third_party/kineto' 2024-06-26T07:02:38.4491352Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:38.4519067Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:38.4547217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:38.4574528Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:38.4603292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:38.4630161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:38.4660136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:38.4687836Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:38.4715944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:38.4744723Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:38.4773798Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:38.4801246Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:38.4827482Z Entering 'third_party/mimalloc' 2024-06-26T07:02:38.4854221Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:38.4883382Z Entering 'third_party/nlohmann' 2024-06-26T07:02:38.4912122Z Entering 'third_party/onnx' 2024-06-26T07:02:38.4952832Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:38.4981650Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:38.5011764Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:38.5041359Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:38.5068312Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:38.5096255Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:38.5121433Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:38.5148146Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:38.5175956Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:38.5201340Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:38.5228232Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:38.5257700Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:38.5285624Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:38.5328171Z Entering 'third_party/pocketfft' 2024-06-26T07:02:38.5357144Z Entering 'third_party/protobuf' 2024-06-26T07:02:38.5385921Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:38.5412538Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:38.5441679Z Entering 'third_party/psimd' 2024-06-26T07:02:38.5469625Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:38.5497617Z Entering 'third_party/pybind11' 2024-06-26T07:02:38.5526005Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:38.5552577Z Entering 'third_party/sleef' 2024-06-26T07:02:38.5582234Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:38.5608713Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:38.5636029Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:38.5662754Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:38.5689262Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:38.5715358Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:38.5754848Z ##[endgroup] 2024-06-26T07:02:38.5787557Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:02:38.5809622Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:02:38.5981661Z Prepare all required actions 2024-06-26T07:02:38.5982128Z Getting action download info 2024-06-26T07:02:38.7353322Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:02:38.7353912Z env: 2024-06-26T07:02:38.7354198Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:38.7354550Z ##[endgroup] 2024-06-26T07:02:38.7411966Z ##[group]Run set -euo pipefail 2024-06-26T07:02:38.7412403Z set -euo pipefail 2024-06-26T07:02:38.7412799Z function get_ec2_metadata() { 2024-06-26T07:02:38.7413342Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:02:38.7414237Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:02:38.7415005Z  category=$1 2024-06-26T07:02:38.7415546Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:02:38.7416196Z  runner_name_str=i-0467b1cafd1897be0 2024-06-26T07:02:38.7416691Z  if [[ -f /.inarc ]]; then 2024-06-26T07:02:38.7417185Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:02:38.7417756Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:02:38.7418450Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:02:38.7419059Z  else 2024-06-26T07:02:38.7419552Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:02:38.7420130Z  fi 2024-06-26T07:02:38.7420415Z } 2024-06-26T07:02:38.7420778Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:02:38.7421357Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:02:38.7422018Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:02:38.7422597Z echo "system info $(uname -a)" 2024-06-26T07:02:38.7430302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:38.7430820Z env: 2024-06-26T07:02:38.7431104Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:38.7431464Z ##[endgroup] 2024-06-26T07:02:38.7500564Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:02:38.7543882Z instance-id: i-0467b1cafd1897be0 2024-06-26T07:02:38.7583260Z instance-type: g5.4xlarge 2024-06-26T07:02:38.7588598Z system info Linux ip-10-0-15-83.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:02:38.7608866Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:02:38.7609789Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:02:38.7617655Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:38.7618260Z env: 2024-06-26T07:02:38.7618544Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:38.7618895Z ##[endgroup] 2024-06-26T07:02:38.7680170Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:02:38.7680792Z if systemctl is-active --quiet docker; then 2024-06-26T07:02:38.7681339Z  echo "Docker daemon is running..."; 2024-06-26T07:02:38.7681791Z else 2024-06-26T07:02:38.7682284Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:02:38.7682869Z fi 2024-06-26T07:02:38.7690693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:38.7691186Z env: 2024-06-26T07:02:38.7691473Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:38.7691825Z ##[endgroup] 2024-06-26T07:02:38.7726920Z Docker daemon is running... 2024-06-26T07:02:38.7770462Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:02:38.7771029Z with: 2024-06-26T07:02:38.7771296Z shell: bash 2024-06-26T07:02:38.7771601Z timeout_minutes: 5 2024-06-26T07:02:38.7771930Z max_attempts: 3 2024-06-26T07:02:38.7772253Z retry_wait_seconds: 30 2024-06-26T07:02:38.7773787Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:02:38.7775239Z polling_interval_seconds: 1 2024-06-26T07:02:38.7775619Z warning_on_retry: true 2024-06-26T07:02:38.7775982Z continue_on_error: false 2024-06-26T07:02:38.7776322Z env: 2024-06-26T07:02:38.7776592Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:38.7776960Z AWS_RETRY_MODE: standard 2024-06-26T07:02:38.7777316Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:02:38.7777659Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:02:38.7778036Z ##[endgroup] 2024-06-26T07:02:39.6461261Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:39.6462544Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:39.6463551Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:39.6464079Z 2024-06-26T07:02:39.6464221Z Login Succeeded 2024-06-26T07:02:39.8278395Z Command completed after 1 attempt(s). 2024-06-26T07:02:39.8324075Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:39.8324943Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:39.8325606Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:39.8333381Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:39.8333913Z env: 2024-06-26T07:02:39.8334200Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:39.8334563Z ##[endgroup] 2024-06-26T07:02:39.8395122Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:02:39.8395904Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:02:39.8396517Z # shellcheck disable=SC2046 2024-06-26T07:02:39.8396986Z docker stop $(docker ps -q) || true 2024-06-26T07:02:39.8397479Z # Prune all of the docker images 2024-06-26T07:02:39.8397938Z docker system prune -af 2024-06-26T07:02:39.8404887Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:39.8405396Z env: 2024-06-26T07:02:39.8405678Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:39.8406029Z ##[endgroup] 2024-06-26T07:02:39.8650904Z "docker stop" requires at least 1 argument. 2024-06-26T07:02:39.8651801Z See 'docker stop --help'. 2024-06-26T07:02:39.8652119Z 2024-06-26T07:02:39.8652419Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:02:39.8652903Z 2024-06-26T07:02:39.8653059Z Stop one or more running containers 2024-06-26T07:02:39.8807669Z Total reclaimed space: 0B 2024-06-26T07:02:39.8836247Z ##[group]Run set +e 2024-06-26T07:02:39.8836619Z set +e 2024-06-26T07:02:39.8837069Z set -x 2024-06-26T07:02:39.8837365Z  2024-06-26T07:02:39.8837687Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:02:39.8838492Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:02:39.8839574Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:02:39.8840327Z # one is returned at random 2024-06-26T07:02:39.8840874Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:02:39.8841380Z  2024-06-26T07:02:39.8841702Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:02:39.8842325Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:02:39.8843089Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:02:39.8843638Z  2024-06-26T07:02:39.8843965Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:02:39.8844673Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:02:39.8845186Z  exit 1 2024-06-26T07:02:39.8845495Z  fi 2024-06-26T07:02:39.8845784Z fi 2024-06-26T07:02:39.8846169Z  2024-06-26T07:02:39.8846537Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:02:39.8847053Z  # Clean up any old records first 2024-06-26T07:02:39.8847573Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:02:39.8848030Z fi 2024-06-26T07:02:39.8848304Z  2024-06-26T07:02:39.8848744Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:02:39.8849290Z cat /etc/hosts 2024-06-26T07:02:39.8857018Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:39.8857509Z env: 2024-06-26T07:02:39.8857785Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:39.8858136Z ##[endgroup] 2024-06-26T07:02:39.8878154Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:02:39.8881404Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:02:39.8882121Z ++ tail -n1 2024-06-26T07:02:39.9229771Z + RESOLVED_IP=18.160.10.36 2024-06-26T07:02:39.9230234Z + '[' -z 18.160.10.36 ']' 2024-06-26T07:02:39.9230695Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:02:39.9235841Z 18.67.76.37 download.pytorch.org 2024-06-26T07:02:39.9236410Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:02:39.9332694Z + echo '18.160.10.36 download.pytorch.org' 2024-06-26T07:02:39.9333189Z + sudo tee -a /etc/hosts 2024-06-26T07:02:39.9632720Z 18.160.10.36 download.pytorch.org 2024-06-26T07:02:39.9646642Z + cat /etc/hosts 2024-06-26T07:02:39.9651059Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:02:39.9662699Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:02:39.9663208Z 18.160.10.36 download.pytorch.org 2024-06-26T07:02:39.9876335Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:02:39.9876948Z with: 2024-06-26T07:02:39.9878043Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:39.9879255Z docker-build-dir: .ci/docker 2024-06-26T07:02:39.9879630Z working-directory: . 2024-06-26T07:02:39.9880099Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:39.9880628Z force-push: false 2024-06-26T07:02:39.9880920Z env: 2024-06-26T07:02:39.9881187Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:39.9881524Z ##[endgroup] 2024-06-26T07:02:39.9898609Z ##[group]Run set -ex 2024-06-26T07:02:39.9898983Z set -ex 2024-06-26T07:02:39.9899273Z  2024-06-26T07:02:39.9899834Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:02:39.9900820Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:02:39.9901748Z # job could then download the pre-built image as usual 2024-06-26T07:02:39.9902503Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:02:39.9903185Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:39.9903832Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:39.9904400Z  2024-06-26T07:02:39.9904920Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:02:39.9905550Z  exit 0 2024-06-26T07:02:39.9905856Z else 2024-06-26T07:02:39.9906232Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:39.9906684Z fi 2024-06-26T07:02:39.9906964Z  2024-06-26T07:02:39.9907447Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:02:39.9908307Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:02:39.9909104Z  # use it as it is, but first let's extract the tag 2024-06-26T07:02:39.9909823Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:02:39.9910558Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:39.9911269Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:39.9911828Z else 2024-06-26T07:02:39.9912274Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:02:39.9912953Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:39.9914013Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:39.9914763Z fi 2024-06-26T07:02:39.9922514Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:39.9923019Z env: 2024-06-26T07:02:39.9923300Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:39.9923648Z REPO_NAME: pytorch 2024-06-26T07:02:39.9924887Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:39.9926087Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:02:39.9926584Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:39.9927096Z ##[endgroup] 2024-06-26T07:02:39.9946689Z + [[ ! -d .ci/docker ]] 2024-06-26T07:02:39.9947120Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:02:39.9947506Z + echo skip=false 2024-06-26T07:02:39.9949519Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:02:39.9952520Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:39.9953901Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:02:39.9959637Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:39.9960283Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:39.9961993Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:39.9988319Z ##[group]Run set +e 2024-06-26T07:02:39.9988685Z set +e 2024-06-26T07:02:39.9988984Z set -x 2024-06-26T07:02:39.9989269Z  2024-06-26T07:02:39.9989549Z login() { 2024-06-26T07:02:39.9990221Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:02:39.9990941Z } 2024-06-26T07:02:39.9991209Z  2024-06-26T07:02:39.9991599Z retry () { 2024-06-26T07:02:39.9991996Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:02:39.9992441Z } 2024-06-26T07:02:39.9992712Z  2024-06-26T07:02:39.9993031Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:02:39.9993450Z  2024-06-26T07:02:39.9994238Z # Check if image already exists, if it does then skip building it 2024-06-26T07:02:39.9994959Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:02:39.9995464Z  exit 0 2024-06-26T07:02:39.9995762Z fi 2024-06-26T07:02:39.9996033Z  2024-06-26T07:02:39.9996534Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:02:39.9997419Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:02:39.9998165Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:02:39.9998838Z  # if we're on the base branch then use the parent commit 2024-06-26T07:02:39.9999444Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:02:39.9999870Z else 2024-06-26T07:02:40.0000339Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:02:40.0001025Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:02:40.0001532Z fi 2024-06-26T07:02:40.0001811Z  2024-06-26T07:02:40.0002132Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:02:40.0002632Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:40.0003088Z  2024-06-26T07:02:40.0003741Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:02:40.0004652Z  exit 0 2024-06-26T07:02:40.0004946Z fi 2024-06-26T07:02:40.0005223Z  2024-06-26T07:02:40.0005661Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:02:40.0006632Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:02:40.0007477Z  exit 1 2024-06-26T07:02:40.0007778Z fi 2024-06-26T07:02:40.0008058Z  2024-06-26T07:02:40.0008545Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:02:40.0009483Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:02:40.0010350Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:02:40.0011327Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:02:40.0012428Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:02:40.0013180Z fi 2024-06-26T07:02:40.0013463Z  2024-06-26T07:02:40.0013827Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:40.0021562Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:40.0022091Z env: 2024-06-26T07:02:40.0022370Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:40.0022752Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:02:40.0023268Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:40.0024585Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:40.0025842Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:40.0026450Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.0027016Z ##[endgroup] 2024-06-26T07:02:40.0047143Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.0047812Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.0048686Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:02:40.0049640Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.4131605Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:40.4132424Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:40.4133247Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:40.4133765Z 2024-06-26T07:02:40.4133897Z Login Succeeded 2024-06-26T07:02:40.4143414Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:40.6081844Z { 2024-06-26T07:02:40.6082246Z "schemaVersion": 2, 2024-06-26T07:02:40.6082915Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:02:40.6083508Z "config": { 2024-06-26T07:02:40.6083969Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:02:40.6084679Z "size": 48681, 2024-06-26T07:02:40.6085229Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:02:40.6085932Z }, 2024-06-26T07:02:40.6086244Z "layers": [ 2024-06-26T07:02:40.6086585Z { 2024-06-26T07:02:40.6087199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6087913Z "size": 28580681, 2024-06-26T07:02:40.6088476Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:02:40.6089113Z }, 2024-06-26T07:02:40.6089365Z { 2024-06-26T07:02:40.6089857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6090514Z "size": 7943451, 2024-06-26T07:02:40.6091052Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:02:40.6091668Z }, 2024-06-26T07:02:40.6091919Z { 2024-06-26T07:02:40.6092350Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6092918Z "size": 55728572, 2024-06-26T07:02:40.6093550Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:02:40.6094517Z }, 2024-06-26T07:02:40.6094888Z { 2024-06-26T07:02:40.6095487Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6096254Z "size": 186, 2024-06-26T07:02:40.6097008Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:02:40.6097867Z }, 2024-06-26T07:02:40.6098237Z { 2024-06-26T07:02:40.6098866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6099709Z "size": 6886, 2024-06-26T07:02:40.6100498Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:02:40.6101467Z }, 2024-06-26T07:02:40.6102218Z + exit 0 2024-06-26T07:02:40.6102610Z { 2024-06-26T07:02:40.6103271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6104135Z "size": 1329001756, 2024-06-26T07:02:40.6105009Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:02:40.6105977Z }, 2024-06-26T07:02:40.6106351Z { 2024-06-26T07:02:40.6107012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6107835Z "size": 62501, 2024-06-26T07:02:40.6108416Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:02:40.6109036Z }, 2024-06-26T07:02:40.6109283Z { 2024-06-26T07:02:40.6109730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6110295Z "size": 1684, 2024-06-26T07:02:40.6110834Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:02:40.6111442Z }, 2024-06-26T07:02:40.6111706Z { 2024-06-26T07:02:40.6112148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6112859Z "size": 1523, 2024-06-26T07:02:40.6113423Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:02:40.6114357Z }, 2024-06-26T07:02:40.6114615Z { 2024-06-26T07:02:40.6115062Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6115626Z "size": 2528295403, 2024-06-26T07:02:40.6116204Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:02:40.6116830Z }, 2024-06-26T07:02:40.6117078Z { 2024-06-26T07:02:40.6117531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6118093Z "size": 86016, 2024-06-26T07:02:40.6118655Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:02:40.6119275Z }, 2024-06-26T07:02:40.6119529Z { 2024-06-26T07:02:40.6119980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6120547Z "size": 1823, 2024-06-26T07:02:40.6121091Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:02:40.6121698Z }, 2024-06-26T07:02:40.6121952Z { 2024-06-26T07:02:40.6122387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6122952Z "size": 246737542, 2024-06-26T07:02:40.6123513Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:02:40.6124230Z }, 2024-06-26T07:02:40.6124486Z { 2024-06-26T07:02:40.6124928Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6125484Z "size": 545, 2024-06-26T07:02:40.6126039Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:02:40.6126657Z }, 2024-06-26T07:02:40.6126912Z { 2024-06-26T07:02:40.6127357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6127913Z "size": 1283, 2024-06-26T07:02:40.6128445Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:02:40.6129050Z }, 2024-06-26T07:02:40.6129308Z { 2024-06-26T07:02:40.6129736Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6130288Z "size": 485, 2024-06-26T07:02:40.6130832Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:02:40.6131454Z }, 2024-06-26T07:02:40.6131696Z { 2024-06-26T07:02:40.6132129Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6132689Z "size": 91710937, 2024-06-26T07:02:40.6133269Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:02:40.6133892Z }, 2024-06-26T07:02:40.6134142Z { 2024-06-26T07:02:40.6134576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6135234Z "size": 3233, 2024-06-26T07:02:40.6135800Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:02:40.6136426Z }, 2024-06-26T07:02:40.6136677Z { 2024-06-26T07:02:40.6137154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6137708Z "size": 1905, 2024-06-26T07:02:40.6138257Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:02:40.6138882Z }, 2024-06-26T07:02:40.6139125Z { 2024-06-26T07:02:40.6139560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6140116Z "size": 702, 2024-06-26T07:02:40.6140659Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:40.6141269Z }, 2024-06-26T07:02:40.6141517Z { 2024-06-26T07:02:40.6141953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6142504Z "size": 2784789002, 2024-06-26T07:02:40.6143075Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:02:40.6143771Z }, 2024-06-26T07:02:40.6144020Z { 2024-06-26T07:02:40.6144446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6145001Z "size": 381, 2024-06-26T07:02:40.6145544Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:02:40.6146165Z }, 2024-06-26T07:02:40.6146409Z { 2024-06-26T07:02:40.6146843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6147400Z "size": 11879, 2024-06-26T07:02:40.6147932Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:02:40.6148538Z }, 2024-06-26T07:02:40.6148791Z { 2024-06-26T07:02:40.6149224Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6149773Z "size": 803, 2024-06-26T07:02:40.6150322Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:02:40.6150945Z }, 2024-06-26T07:02:40.6151194Z { 2024-06-26T07:02:40.6151625Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6152181Z "size": 106, 2024-06-26T07:02:40.6152713Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:02:40.6153316Z }, 2024-06-26T07:02:40.6153557Z { 2024-06-26T07:02:40.6154212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6154782Z "size": 503, 2024-06-26T07:02:40.6155319Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:02:40.6155933Z }, 2024-06-26T07:02:40.6156187Z { 2024-06-26T07:02:40.6156625Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6157189Z "size": 121477177, 2024-06-26T07:02:40.6157764Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:02:40.6158373Z }, 2024-06-26T07:02:40.6158630Z { 2024-06-26T07:02:40.6159063Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6159626Z "size": 109, 2024-06-26T07:02:40.6160172Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:02:40.6160790Z }, 2024-06-26T07:02:40.6161036Z { 2024-06-26T07:02:40.6161476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6162035Z "size": 491, 2024-06-26T07:02:40.6162568Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:02:40.6163178Z }, 2024-06-26T07:02:40.6163436Z { 2024-06-26T07:02:40.6163874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6164526Z "size": 291, 2024-06-26T07:02:40.6165081Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:02:40.6165796Z }, 2024-06-26T07:02:40.6166056Z { 2024-06-26T07:02:40.6166493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6167053Z "size": 101, 2024-06-26T07:02:40.6167595Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:02:40.6168253Z }, 2024-06-26T07:02:40.6168506Z { 2024-06-26T07:02:40.6168944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6169511Z "size": 1473, 2024-06-26T07:02:40.6170042Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:02:40.6170644Z }, 2024-06-26T07:02:40.6170899Z { 2024-06-26T07:02:40.6171346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6171901Z "size": 446710781, 2024-06-26T07:02:40.6172468Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:02:40.6173089Z }, 2024-06-26T07:02:40.6173345Z { 2024-06-26T07:02:40.6173784Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6174471Z "size": 161, 2024-06-26T07:02:40.6174998Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:02:40.6175592Z }, 2024-06-26T07:02:40.6175848Z { 2024-06-26T07:02:40.6176290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6176847Z "size": 564, 2024-06-26T07:02:40.6177386Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:02:40.6178009Z }, 2024-06-26T07:02:40.6178265Z { 2024-06-26T07:02:40.6178706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6179258Z "size": 35874600, 2024-06-26T07:02:40.6179825Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:02:40.6180443Z }, 2024-06-26T07:02:40.6180698Z { 2024-06-26T07:02:40.6181142Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6181706Z "size": 104, 2024-06-26T07:02:40.6182235Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:02:40.6182836Z }, 2024-06-26T07:02:40.6183088Z { 2024-06-26T07:02:40.6183524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6184078Z "size": 425, 2024-06-26T07:02:40.6184604Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:02:40.6185216Z }, 2024-06-26T07:02:40.6185469Z { 2024-06-26T07:02:40.6185898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6186459Z "size": 20262084, 2024-06-26T07:02:40.6187027Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:02:40.6187700Z }, 2024-06-26T07:02:40.6187947Z { 2024-06-26T07:02:40.6188387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6188941Z "size": 439, 2024-06-26T07:02:40.6189474Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:02:40.6190072Z }, 2024-06-26T07:02:40.6190327Z { 2024-06-26T07:02:40.6190763Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6191318Z "size": 702, 2024-06-26T07:02:40.6191860Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:40.6192485Z }, 2024-06-26T07:02:40.6192740Z { 2024-06-26T07:02:40.6193173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6193901Z "size": 143, 2024-06-26T07:02:40.6194477Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:02:40.6195086Z }, 2024-06-26T07:02:40.6195328Z { 2024-06-26T07:02:40.6195764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6196419Z "size": 135, 2024-06-26T07:02:40.6196972Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:02:40.6197644Z }, 2024-06-26T07:02:40.6197896Z { 2024-06-26T07:02:40.6198392Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6198949Z "size": 4911495247, 2024-06-26T07:02:40.6199505Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:02:40.6200106Z }, 2024-06-26T07:02:40.6200350Z { 2024-06-26T07:02:40.6200783Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6201337Z "size": 190, 2024-06-26T07:02:40.6201878Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:02:40.6202498Z }, 2024-06-26T07:02:40.6202749Z { 2024-06-26T07:02:40.6203182Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6203737Z "size": 565, 2024-06-26T07:02:40.6204415Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:02:40.6205109Z }, 2024-06-26T07:02:40.6205362Z { 2024-06-26T07:02:40.6205794Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6206356Z "size": 43163095, 2024-06-26T07:02:40.6206922Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:02:40.6207536Z }, 2024-06-26T07:02:40.6207792Z { 2024-06-26T07:02:40.6208234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6208793Z "size": 106, 2024-06-26T07:02:40.6209332Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:02:40.6209955Z }, 2024-06-26T07:02:40.6210204Z { 2024-06-26T07:02:40.6210646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6211217Z "size": 1213, 2024-06-26T07:02:40.6211764Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:02:40.6212376Z }, 2024-06-26T07:02:40.6212627Z { 2024-06-26T07:02:40.6213067Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6213627Z "size": 702, 2024-06-26T07:02:40.6214176Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:40.6214790Z }, 2024-06-26T07:02:40.6215043Z { 2024-06-26T07:02:40.6215484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6216040Z "size": 138, 2024-06-26T07:02:40.6216582Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:02:40.6217210Z }, 2024-06-26T07:02:40.6217459Z { 2024-06-26T07:02:40.6217898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6218450Z "size": 120, 2024-06-26T07:02:40.6219009Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:02:40.6219630Z }, 2024-06-26T07:02:40.6219875Z { 2024-06-26T07:02:40.6220317Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6220880Z "size": 1904157495, 2024-06-26T07:02:40.6221457Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:02:40.6222078Z }, 2024-06-26T07:02:40.6222332Z { 2024-06-26T07:02:40.6222774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6223328Z "size": 172, 2024-06-26T07:02:40.6223869Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:02:40.6224490Z }, 2024-06-26T07:02:40.6224740Z { 2024-06-26T07:02:40.6225177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6225729Z "size": 908, 2024-06-26T07:02:40.6226284Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:02:40.6226983Z }, 2024-06-26T07:02:40.6227243Z { 2024-06-26T07:02:40.6227718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6228280Z "size": 702, 2024-06-26T07:02:40.6228826Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:40.6229438Z }, 2024-06-26T07:02:40.6229689Z { 2024-06-26T07:02:40.6230126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6230682Z "size": 134, 2024-06-26T07:02:40.6231216Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:02:40.6231837Z }, 2024-06-26T07:02:40.6232093Z { 2024-06-26T07:02:40.6232524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6233081Z "size": 32, 2024-06-26T07:02:40.6233625Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:40.6234410Z }, 2024-06-26T07:02:40.6234661Z { 2024-06-26T07:02:40.6235100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6235762Z "size": 156, 2024-06-26T07:02:40.6236312Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:02:40.6236932Z }, 2024-06-26T07:02:40.6237183Z { 2024-06-26T07:02:40.6237632Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6238225Z "size": 1841, 2024-06-26T07:02:40.6238768Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:02:40.6239388Z }, 2024-06-26T07:02:40.6239635Z { 2024-06-26T07:02:40.6240067Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6240627Z "size": 7529784, 2024-06-26T07:02:40.6241184Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:02:40.6241799Z }, 2024-06-26T07:02:40.6242042Z { 2024-06-26T07:02:40.6242487Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6243042Z "size": 165, 2024-06-26T07:02:40.6243590Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:02:40.6244294Z }, 2024-06-26T07:02:40.6244545Z { 2024-06-26T07:02:40.6244977Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6245530Z "size": 7944, 2024-06-26T07:02:40.6246071Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:02:40.6246684Z }, 2024-06-26T07:02:40.6246934Z { 2024-06-26T07:02:40.6247361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6247915Z "size": 8070, 2024-06-26T07:02:40.6248452Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:02:40.6249058Z }, 2024-06-26T07:02:40.6249300Z { 2024-06-26T07:02:40.6249749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6250307Z "size": 303, 2024-06-26T07:02:40.6250856Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:02:40.6251467Z }, 2024-06-26T07:02:40.6251725Z { 2024-06-26T07:02:40.6252159Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6252706Z "size": 7629071, 2024-06-26T07:02:40.6253232Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:02:40.6253818Z }, 2024-06-26T07:02:40.6254068Z { 2024-06-26T07:02:40.6254494Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6255051Z "size": 108, 2024-06-26T07:02:40.6255601Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:02:40.6256219Z }, 2024-06-26T07:02:40.6256462Z { 2024-06-26T07:02:40.6256901Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6257542Z "size": 54145775, 2024-06-26T07:02:40.6258112Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:02:40.6258729Z }, 2024-06-26T07:02:40.6258977Z { 2024-06-26T07:02:40.6259413Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6259961Z "size": 474, 2024-06-26T07:02:40.6260495Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:02:40.6261106Z }, 2024-06-26T07:02:40.6261351Z { 2024-06-26T07:02:40.6261777Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6262337Z "size": 1374858747, 2024-06-26T07:02:40.6262890Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:02:40.6263484Z }, 2024-06-26T07:02:40.6263726Z { 2024-06-26T07:02:40.6264159Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6264708Z "size": 106, 2024-06-26T07:02:40.6265257Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:02:40.6265942Z }, 2024-06-26T07:02:40.6266194Z { 2024-06-26T07:02:40.6266626Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6267222Z "size": 558, 2024-06-26T07:02:40.6267767Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:02:40.6268389Z }, 2024-06-26T07:02:40.6268637Z { 2024-06-26T07:02:40.6269063Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6269619Z "size": 46248513, 2024-06-26T07:02:40.6270170Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:02:40.6270773Z }, 2024-06-26T07:02:40.6271014Z { 2024-06-26T07:02:40.6271449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6272001Z "size": 111, 2024-06-26T07:02:40.6272534Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:02:40.6273145Z }, 2024-06-26T07:02:40.6273400Z { 2024-06-26T07:02:40.6274004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6274564Z "size": 32, 2024-06-26T07:02:40.6275108Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:40.6275734Z }, 2024-06-26T07:02:40.6275983Z { 2024-06-26T07:02:40.6276410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6276969Z "size": 32, 2024-06-26T07:02:40.6277506Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:40.6278115Z }, 2024-06-26T07:02:40.6278374Z { 2024-06-26T07:02:40.6278811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6279361Z "size": 32, 2024-06-26T07:02:40.6279901Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:40.6280522Z }, 2024-06-26T07:02:40.6280770Z { 2024-06-26T07:02:40.6281208Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:40.6281754Z "size": 32, 2024-06-26T07:02:40.6282288Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:40.6282903Z } 2024-06-26T07:02:40.6283152Z ] 2024-06-26T07:02:40.6283392Z } 2024-06-26T07:02:40.6399804Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:02:40.6400300Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:02:40.6400857Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:02:40.6409014Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:40.6409516Z env: 2024-06-26T07:02:40.6409800Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:40.6410958Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:40.6412129Z ##[endgroup] 2024-06-26T07:02:40.6435764Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:40.6485533Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:02:40.6486111Z with: 2024-06-26T07:02:40.6487188Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:40.6488511Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.6489033Z env: 2024-06-26T07:02:40.6489313Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:40.6489669Z ##[endgroup] 2024-06-26T07:02:40.6504800Z ##[group]Run set -x 2024-06-26T07:02:40.6505140Z set -x 2024-06-26T07:02:40.6505438Z set +e 2024-06-26T07:02:40.6505729Z  2024-06-26T07:02:40.6505999Z login() { 2024-06-26T07:02:40.6506661Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:02:40.6507543Z } 2024-06-26T07:02:40.6507822Z  2024-06-26T07:02:40.6508121Z retry () { 2024-06-26T07:02:40.6508514Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:02:40.6508957Z } 2024-06-26T07:02:40.6509238Z  2024-06-26T07:02:40.6509555Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:02:40.6509969Z  2024-06-26T07:02:40.6510243Z set -e 2024-06-26T07:02:40.6510720Z # ignore output since only exit code is used for conditional 2024-06-26T07:02:40.6511430Z # only pull docker image if it's not available locally 2024-06-26T07:02:40.6512198Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:02:40.6512907Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:02:40.6513351Z fi 2024-06-26T07:02:40.6521184Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:40.6521683Z env: 2024-06-26T07:02:40.6521963Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:40.6523105Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:40.6524504Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.6525020Z ##[endgroup] 2024-06-26T07:02:40.6545477Z + set +e 2024-06-26T07:02:40.6546730Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.6547595Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:40.6548290Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:02:40.6549061Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:41.0642722Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:41.0643600Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:41.0644920Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:41.0645465Z 2024-06-26T07:02:41.0645608Z Login Succeeded 2024-06-26T07:02:41.0653609Z + set -e 2024-06-26T07:02:41.0654905Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:41.0781933Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:41.0784058Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:41.2789210Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:02:41.2790566Z 7a2c55901189: Pulling fs layer 2024-06-26T07:02:41.2791114Z 224fe954d725: Pulling fs layer 2024-06-26T07:02:41.2791968Z 75722010b82e: Pulling fs layer 2024-06-26T07:02:41.2792556Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:02:41.2793063Z b57676e46aee: Pulling fs layer 2024-06-26T07:02:41.2793621Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:02:41.2794423Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:02:41.2794948Z 0c1227890755: Pulling fs layer 2024-06-26T07:02:41.2795474Z d8d1234baab3: Pulling fs layer 2024-06-26T07:02:41.2796002Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:02:41.2796527Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:02:41.2797050Z 309b872da805: Pulling fs layer 2024-06-26T07:02:41.2797576Z e82932cb002c: Pulling fs layer 2024-06-26T07:02:41.2797977Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:02:41.2798358Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:02:41.2798741Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:02:41.2799112Z f4863801e30c: Pulling fs layer 2024-06-26T07:02:41.2799621Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:02:41.2800001Z cf2f34f225db: Pulling fs layer 2024-06-26T07:02:41.2800384Z df546d7b3e34: Pulling fs layer 2024-06-26T07:02:41.2800809Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:02:41.2801177Z 96d0e2032184: Pulling fs layer 2024-06-26T07:02:41.2801548Z 06e12ce59790: Pulling fs layer 2024-06-26T07:02:41.2801922Z ed712f63cde7: Pulling fs layer 2024-06-26T07:02:41.2802298Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:02:41.2802671Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:02:41.2803049Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:02:41.2803424Z de45a14c9b22: Pulling fs layer 2024-06-26T07:02:41.2803792Z e99241e77209: Pulling fs layer 2024-06-26T07:02:41.2804338Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:02:41.2804703Z d527cbbb87e3: Waiting 2024-06-26T07:02:41.2805044Z a823ffa4c541: Pulling fs layer 2024-06-26T07:02:41.2805416Z 44ba272eae20: Pulling fs layer 2024-06-26T07:02:41.2805776Z a8c1e85b5e14: Waiting 2024-06-26T07:02:41.2806120Z da02d278e524: Pulling fs layer 2024-06-26T07:02:41.2806496Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:02:41.2806855Z b57676e46aee: Waiting 2024-06-26T07:02:41.2807187Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:02:41.2807616Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:02:41.2807984Z b2950a036534: Pulling fs layer 2024-06-26T07:02:41.2808340Z a41a8d1c11c8: Waiting 2024-06-26T07:02:41.2808672Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:02:41.2809046Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:02:41.2809397Z 16d76a79d5d6: Waiting 2024-06-26T07:02:41.2809711Z f4863801e30c: Waiting 2024-06-26T07:02:41.2810041Z 7a868239227f: Pulling fs layer 2024-06-26T07:02:41.2810404Z 4bdf41c9ecbe: Waiting 2024-06-26T07:02:41.2810716Z 3d914950f8ad: Waiting 2024-06-26T07:02:41.2811025Z 0c1227890755: Waiting 2024-06-26T07:02:41.2811339Z d37dd9ed8948: Waiting 2024-06-26T07:02:41.2811666Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:02:41.2812030Z ca9dd73e3e93: Waiting 2024-06-26T07:02:41.2812349Z cf2f34f225db: Waiting 2024-06-26T07:02:41.2812751Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:02:41.2813248Z df546d7b3e34: Waiting 2024-06-26T07:02:41.2813752Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:02:41.2814280Z e82932cb002c: Waiting 2024-06-26T07:02:41.2814716Z 96d0e2032184: Waiting 2024-06-26T07:02:41.2815095Z 309b872da805: Waiting 2024-06-26T07:02:41.2815424Z c1068d03a811: Pulling fs layer 2024-06-26T07:02:41.2815782Z ec1e7978c1fe: Waiting 2024-06-26T07:02:41.2816095Z d8d1234baab3: Waiting 2024-06-26T07:02:41.2816424Z 671b4a301d81: Pulling fs layer 2024-06-26T07:02:41.2816784Z 7ed32bc8e469: Waiting 2024-06-26T07:02:41.2817113Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:02:41.2817488Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:02:41.2817868Z fb2e97768560: Pulling fs layer 2024-06-26T07:02:41.2818243Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:02:41.2818609Z e08e88c13668: Pulling fs layer 2024-06-26T07:02:41.2818968Z ed712f63cde7: Waiting 2024-06-26T07:02:41.2819298Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:02:41.2819663Z 35e8f4e2a42f: Waiting 2024-06-26T07:02:41.2819971Z a823ffa4c541: Waiting 2024-06-26T07:02:41.2820286Z 44ba272eae20: Waiting 2024-06-26T07:02:41.2820719Z de45a14c9b22: Waiting 2024-06-26T07:02:41.2821063Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:02:41.2821440Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:02:41.2821814Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:02:41.2822188Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:02:41.2822558Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:02:41.2822940Z e797fe86caef: Pulling fs layer 2024-06-26T07:02:41.2823312Z 967190076fc0: Pulling fs layer 2024-06-26T07:02:41.2823684Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:02:41.2824050Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:02:41.2824423Z cc91933e985a: Pulling fs layer 2024-06-26T07:02:41.2824792Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:02:41.2825158Z 526083994857: Pulling fs layer 2024-06-26T07:02:41.2825518Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:02:41.2825893Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:02:41.2826327Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:02:41.2826690Z 95237e7311a1: Pulling fs layer 2024-06-26T07:02:41.2827092Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:02:41.2827613Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:02:41.2828156Z 2060028f7488: Pulling fs layer 2024-06-26T07:02:41.2828579Z b2950a036534: Waiting 2024-06-26T07:02:41.2828911Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:02:41.2829269Z 25cd0c482ad6: Waiting 2024-06-26T07:02:41.2829586Z 9cd4c745f8e4: Waiting 2024-06-26T07:02:41.2829894Z 9bf94bc45658: Waiting 2024-06-26T07:02:41.2830204Z 7a868239227f: Waiting 2024-06-26T07:02:41.2830513Z 967190076fc0: Waiting 2024-06-26T07:02:41.2830815Z 95237e7311a1: Waiting 2024-06-26T07:02:41.2831124Z 83911d7c7ecf: Waiting 2024-06-26T07:02:41.2831439Z b3c5d66c679f: Waiting 2024-06-26T07:02:41.2831751Z 526083994857: Waiting 2024-06-26T07:02:41.2832054Z 789eff6c51aa: Waiting 2024-06-26T07:02:41.2832370Z 05a6f6dcc1c0: Waiting 2024-06-26T07:02:41.2832682Z 60156b40fcbe: Waiting 2024-06-26T07:02:41.2832999Z de0f8a8c45c6: Waiting 2024-06-26T07:02:41.2833308Z 2060028f7488: Waiting 2024-06-26T07:02:41.2833621Z 6bc9dc8315b0: Waiting 2024-06-26T07:02:41.2834082Z 4fb0a83f0685: Waiting 2024-06-26T07:02:41.2834387Z 9f3187916e7d: Waiting 2024-06-26T07:02:41.2834695Z 25e19d81d9fc: Waiting 2024-06-26T07:02:41.2835007Z 236c8f0245f4: Waiting 2024-06-26T07:02:41.2835310Z 84b2c7a89982: Waiting 2024-06-26T07:02:41.2835621Z c1068d03a811: Waiting 2024-06-26T07:02:41.2835930Z 671b4a301d81: Waiting 2024-06-26T07:02:41.2836239Z 67760cf5ec46: Waiting 2024-06-26T07:02:41.2836547Z ddc9740bcea2: Waiting 2024-06-26T07:02:41.2836864Z fb2e97768560: Waiting 2024-06-26T07:02:41.2837215Z e412c7f3b2de: Waiting 2024-06-26T07:02:41.2837535Z e99241e77209: Waiting 2024-06-26T07:02:41.2837838Z e08e88c13668: Waiting 2024-06-26T07:02:41.2838152Z e797fe86caef: Waiting 2024-06-26T07:02:41.2838465Z 4f4fb700ef54: Waiting 2024-06-26T07:02:41.2838775Z 3a778ddfac2a: Waiting 2024-06-26T07:02:41.2839091Z dda1ce302b9a: Waiting 2024-06-26T07:02:41.2839409Z 12ba7431ff38: Waiting 2024-06-26T07:02:41.2839724Z 1b0e27c35f07: Waiting 2024-06-26T07:02:41.2840040Z cf7c2cb896cb: Waiting 2024-06-26T07:02:41.4186099Z 224fe954d725: Verifying Checksum 2024-06-26T07:02:41.4186693Z 224fe954d725: Download complete 2024-06-26T07:02:41.5086294Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:02:41.5086896Z d527cbbb87e3: Download complete 2024-06-26T07:02:41.5901573Z b57676e46aee: Download complete 2024-06-26T07:02:41.6268547Z 7a2c55901189: Verifying Checksum 2024-06-26T07:02:41.6268977Z 7a2c55901189: Download complete 2024-06-26T07:02:41.7041724Z a41a8d1c11c8: Verifying Checksum 2024-06-26T07:02:41.7042457Z a41a8d1c11c8: Download complete 2024-06-26T07:02:41.7794471Z 0c1227890755: Verifying Checksum 2024-06-26T07:02:41.7795062Z 0c1227890755: Download complete 2024-06-26T07:02:41.8728465Z d8d1234baab3: Verifying Checksum 2024-06-26T07:02:41.8729014Z d8d1234baab3: Download complete 2024-06-26T07:02:41.9074028Z 75722010b82e: Verifying Checksum 2024-06-26T07:02:41.9074465Z 75722010b82e: Download complete 2024-06-26T07:02:41.9966330Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:02:41.9967230Z ec1e7978c1fe: Download complete 2024-06-26T07:02:42.0686641Z 309b872da805: Verifying Checksum 2024-06-26T07:02:42.0687067Z 309b872da805: Download complete 2024-06-26T07:02:42.6709124Z 7a2c55901189: Pull complete 2024-06-26T07:02:42.9943002Z 224fe954d725: Pull complete 2024-06-26T07:02:43.9517452Z 75722010b82e: Pull complete 2024-06-26T07:02:43.9977225Z d527cbbb87e3: Pull complete 2024-06-26T07:02:44.0442831Z b57676e46aee: Pull complete 2024-06-26T07:02:44.7574432Z e82932cb002c: Verifying Checksum 2024-06-26T07:02:44.7574869Z e82932cb002c: Download complete 2024-06-26T07:02:44.8324691Z ca9dd73e3e93: Download complete 2024-06-26T07:02:44.9097334Z 16d76a79d5d6: Download complete 2024-06-26T07:02:45.0019044Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:02:45.0019480Z 4bdf41c9ecbe: Download complete 2024-06-26T07:02:45.9725230Z f4863801e30c: Verifying Checksum 2024-06-26T07:02:45.9726153Z f4863801e30c: Download complete 2024-06-26T07:02:46.0436062Z d37dd9ed8948: Download complete 2024-06-26T07:02:46.1301119Z cf2f34f225db: Verifying Checksum 2024-06-26T07:02:46.1301552Z cf2f34f225db: Download complete 2024-06-26T07:02:46.2500052Z df546d7b3e34: Verifying Checksum 2024-06-26T07:02:46.2500463Z df546d7b3e34: Download complete 2024-06-26T07:02:54.9599643Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:02:54.9600300Z a8c1e85b5e14: Download complete 2024-06-26T07:02:55.0438816Z 96d0e2032184: Verifying Checksum 2024-06-26T07:02:55.0439427Z 96d0e2032184: Download complete 2024-06-26T07:02:55.1281952Z 06e12ce59790: Download complete 2024-06-26T07:02:55.2016564Z ed712f63cde7: Verifying Checksum 2024-06-26T07:02:55.2017157Z ed712f63cde7: Download complete 2024-06-26T07:02:55.2786759Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:02:55.2787287Z a97a71ef8a4e: Download complete 2024-06-26T07:02:55.3836716Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:02:55.3837236Z 25cd0c482ad6: Download complete 2024-06-26T07:02:56.8689414Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:02:56.8689894Z 9bf94bc45658: Download complete 2024-06-26T07:02:56.9469907Z de45a14c9b22: Download complete 2024-06-26T07:02:57.0209449Z e99241e77209: Verifying Checksum 2024-06-26T07:02:57.0210083Z e99241e77209: Download complete 2024-06-26T07:02:57.0931328Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:02:57.0931877Z 35e8f4e2a42f: Download complete 2024-06-26T07:02:57.1905425Z a823ffa4c541: Verifying Checksum 2024-06-26T07:02:57.1905973Z a823ffa4c541: Download complete 2024-06-26T07:02:57.2723955Z 44ba272eae20: Verifying Checksum 2024-06-26T07:02:57.2724810Z 44ba272eae20: Download complete 2024-06-26T07:03:04.6279313Z da02d278e524: Verifying Checksum 2024-06-26T07:03:04.6279770Z da02d278e524: Download complete 2024-06-26T07:03:04.7048338Z 73a0ce681c29: Download complete 2024-06-26T07:03:04.7723014Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:03:04.7723592Z 49737bd5ae70: Download complete 2024-06-26T07:03:05.2277437Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:03:05.2278058Z 9cd4c745f8e4: Download complete 2024-06-26T07:03:05.3228452Z b2950a036534: Verifying Checksum 2024-06-26T07:03:05.3229120Z b2950a036534: Download complete 2024-06-26T07:03:05.4226315Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:03:05.4226891Z 25e19d81d9fc: Download complete 2024-06-26T07:03:05.8049966Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:03:05.8845783Z 7a868239227f: Verifying Checksum 2024-06-26T07:03:05.8846694Z 7a868239227f: Download complete 2024-06-26T07:03:06.0018151Z 83911d7c7ecf: Download complete 2024-06-26T07:03:06.1069216Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:03:06.1069876Z b3c5d66c679f: Download complete 2024-06-26T07:03:07.2131569Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:03:07.2132044Z 7ed32bc8e469: Download complete 2024-06-26T07:03:07.2997915Z c1068d03a811: Verifying Checksum 2024-06-26T07:03:07.2998404Z c1068d03a811: Download complete 2024-06-26T07:03:07.4072981Z 671b4a301d81: Verifying Checksum 2024-06-26T07:03:07.4073563Z 671b4a301d81: Download complete 2024-06-26T07:03:07.6538130Z a8c1e85b5e14: Pull complete 2024-06-26T07:03:07.8470191Z a41a8d1c11c8: Pull complete 2024-06-26T07:03:07.8960932Z 0c1227890755: Pull complete 2024-06-26T07:03:07.9331819Z d8d1234baab3: Pull complete 2024-06-26T07:03:07.9476421Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:03:07.9476837Z 67760cf5ec46: Download complete 2024-06-26T07:03:08.0307936Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:03:08.0308494Z ddc9740bcea2: Download complete 2024-06-26T07:03:08.0942486Z fb2e97768560: Download complete 2024-06-26T07:03:08.1728950Z b599c5a8c7cb: Download complete 2024-06-26T07:03:08.2509865Z e08e88c13668: Verifying Checksum 2024-06-26T07:03:08.2510306Z e08e88c13668: Download complete 2024-06-26T07:03:37.6584159Z 7ed32bc8e469: Pull complete 2024-06-26T07:03:38.0884202Z ec1e7978c1fe: Pull complete 2024-06-26T07:03:38.6463791Z 309b872da805: Pull complete 2024-06-26T07:03:45.0179964Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:03:45.0180821Z 3d914950f8ad: Download complete 2024-06-26T07:03:45.1071504Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:03:45.1072006Z cf7c2cb896cb: Download complete 2024-06-26T07:03:45.1870424Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:03:45.1870948Z 3a778ddfac2a: Download complete 2024-06-26T07:03:45.2644352Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:03:45.2644917Z 236c8f0245f4: Download complete 2024-06-26T07:03:45.2714442Z 4f4fb700ef54: Download complete 2024-06-26T07:03:45.3513118Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:03:45.3513651Z dda1ce302b9a: Download complete 2024-06-26T07:03:45.4280719Z e797fe86caef: Verifying Checksum 2024-06-26T07:03:45.4281251Z e797fe86caef: Download complete 2024-06-26T07:03:45.7678570Z 967190076fc0: Verifying Checksum 2024-06-26T07:03:45.7679114Z 967190076fc0: Download complete 2024-06-26T07:03:45.8462663Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:03:45.8463314Z 4fb0a83f0685: Download complete 2024-06-26T07:03:45.9325013Z 68af2ec4e30a: Download complete 2024-06-26T07:03:46.0079842Z cc91933e985a: Verifying Checksum 2024-06-26T07:03:46.0080315Z cc91933e985a: Download complete 2024-06-26T07:03:46.1020532Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:03:46.1021031Z 9f3187916e7d: Download complete 2024-06-26T07:03:46.2664823Z e82932cb002c: Pull complete 2024-06-26T07:03:46.3593042Z 526083994857: Verifying Checksum 2024-06-26T07:03:46.3593605Z 526083994857: Download complete 2024-06-26T07:03:46.4340864Z 05a6f6dcc1c0: Download complete 2024-06-26T07:03:46.4926798Z ca9dd73e3e93: Pull complete 2024-06-26T07:03:46.7495073Z 16d76a79d5d6: Pull complete 2024-06-26T07:03:46.9871171Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:03:48.5381826Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:03:48.5382272Z 6bc9dc8315b0: Download complete 2024-06-26T07:03:48.6834108Z 1b0e27c35f07: Download complete 2024-06-26T07:03:49.3661567Z f4863801e30c: Pull complete 2024-06-26T07:03:49.5600972Z d37dd9ed8948: Pull complete 2024-06-26T07:03:49.7918216Z cf2f34f225db: Pull complete 2024-06-26T07:03:50.0370035Z df546d7b3e34: Pull complete 2024-06-26T07:03:52.0353599Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:03:52.0354275Z 12ba7431ff38: Download complete 2024-06-26T07:03:52.1153619Z 789eff6c51aa: Download complete 2024-06-26T07:03:52.2042264Z de0f8a8c45c6: Download complete 2024-06-26T07:03:53.3475267Z 2060028f7488: Verifying Checksum 2024-06-26T07:03:53.3475729Z 2060028f7488: Download complete 2024-06-26T07:03:53.4356512Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:03:53.4357072Z e412c7f3b2de: Download complete 2024-06-26T07:04:15.7727984Z 95237e7311a1: Verifying Checksum 2024-06-26T07:04:15.7728551Z 95237e7311a1: Download complete 2024-06-26T07:04:49.3697850Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:04:49.3698414Z 84b2c7a89982: Download complete 2024-06-26T07:05:13.6960441Z 3d914950f8ad: Pull complete 2024-06-26T07:05:13.8801523Z 96d0e2032184: Pull complete 2024-06-26T07:05:14.0343909Z 06e12ce59790: Pull complete 2024-06-26T07:05:14.1464646Z ed712f63cde7: Pull complete 2024-06-26T07:05:14.3326076Z a97a71ef8a4e: Pull complete 2024-06-26T07:05:14.5532466Z 25cd0c482ad6: Pull complete 2024-06-26T07:05:18.1199464Z 9bf94bc45658: Pull complete 2024-06-26T07:05:18.3651449Z de45a14c9b22: Pull complete 2024-06-26T07:05:18.5963490Z e99241e77209: Pull complete 2024-06-26T07:05:18.8153230Z 35e8f4e2a42f: Pull complete 2024-06-26T07:05:18.9333290Z a823ffa4c541: Pull complete 2024-06-26T07:05:19.1454026Z 44ba272eae20: Pull complete 2024-06-26T07:05:27.5042376Z da02d278e524: Pull complete 2024-06-26T07:05:27.7325877Z 73a0ce681c29: Pull complete 2024-06-26T07:05:27.9701431Z 49737bd5ae70: Pull complete 2024-06-26T07:05:28.9436297Z 9cd4c745f8e4: Pull complete 2024-06-26T07:05:29.1800382Z b2950a036534: Pull complete 2024-06-26T07:05:29.4021722Z 25e19d81d9fc: Pull complete 2024-06-26T07:05:29.8753653Z 60156b40fcbe: Pull complete 2024-06-26T07:05:30.1010994Z 7a868239227f: Pull complete 2024-06-26T07:05:30.5865196Z 83911d7c7ecf: Pull complete 2024-06-26T07:05:30.7870574Z b3c5d66c679f: Pull complete 2024-06-26T07:06:10.1021690Z 84b2c7a89982: Pull complete 2024-06-26T07:06:10.3330546Z c1068d03a811: Pull complete 2024-06-26T07:06:10.5449220Z 671b4a301d81: Pull complete 2024-06-26T07:06:12.6108759Z 67760cf5ec46: Pull complete 2024-06-26T07:06:12.6449480Z ddc9740bcea2: Pull complete 2024-06-26T07:06:12.6963783Z fb2e97768560: Pull complete 2024-06-26T07:06:12.7890050Z b599c5a8c7cb: Pull complete 2024-06-26T07:06:12.8352548Z e08e88c13668: Pull complete 2024-06-26T07:06:47.1570997Z 12ba7431ff38: Pull complete 2024-06-26T07:06:47.3838601Z cf7c2cb896cb: Pull complete 2024-06-26T07:06:47.6076412Z 3a778ddfac2a: Pull complete 2024-06-26T07:06:48.0946416Z 236c8f0245f4: Pull complete 2024-06-26T07:06:48.3067102Z 4f4fb700ef54: Pull complete 2024-06-26T07:06:48.5075756Z dda1ce302b9a: Pull complete 2024-06-26T07:06:48.7249894Z e797fe86caef: Pull complete 2024-06-26T07:06:49.1000461Z 967190076fc0: Pull complete 2024-06-26T07:06:49.3367595Z 4fb0a83f0685: Pull complete 2024-06-26T07:06:49.5800293Z 68af2ec4e30a: Pull complete 2024-06-26T07:06:49.8163181Z cc91933e985a: Pull complete 2024-06-26T07:06:50.0389646Z 9f3187916e7d: Pull complete 2024-06-26T07:06:51.1578873Z 526083994857: Pull complete 2024-06-26T07:06:51.3822201Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:06:53.4779829Z 6bc9dc8315b0: Pull complete 2024-06-26T07:06:53.7102642Z 1b0e27c35f07: Pull complete 2024-06-26T07:07:07.2318795Z 95237e7311a1: Pull complete 2024-06-26T07:07:07.4583538Z 789eff6c51aa: Pull complete 2024-06-26T07:07:07.7068767Z de0f8a8c45c6: Pull complete 2024-06-26T07:07:08.5969392Z 2060028f7488: Pull complete 2024-06-26T07:07:08.8243822Z e412c7f3b2de: Pull complete 2024-06-26T07:07:09.9087063Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:07:09.9516666Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:09.9718472Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:09.9758117Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:07:09.9759049Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:07:09.9767213Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:09.9767712Z env: 2024-06-26T07:07:09.9767995Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:09.9768335Z ##[endgroup] 2024-06-26T07:07:09.9924164Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:07:09.9924850Z with: 2024-06-26T07:07:09.9925143Z driver-version: 550.54.15 2024-06-26T07:07:09.9925482Z env: 2024-06-26T07:07:09.9925766Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:09.9926119Z ##[endgroup] 2024-06-26T07:07:10.0030668Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:07:10.0031252Z with: 2024-06-26T07:07:10.0031532Z timeout_minutes: 10 2024-06-26T07:07:10.0031860Z max_attempts: 3 2024-06-26T07:07:10.0063855Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:07:10.0095951Z retry_wait_seconds: 10 2024-06-26T07:07:10.0096332Z polling_interval_seconds: 1 2024-06-26T07:07:10.0096711Z warning_on_retry: true 2024-06-26T07:07:10.0097071Z continue_on_error: false 2024-06-26T07:07:10.0097416Z env: 2024-06-26T07:07:10.0097694Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:10.0098052Z DRIVER_VERSION: 550.54.15 2024-06-26T07:07:10.0098411Z ##[endgroup] 2024-06-26T07:07:10.0621236Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:07:10.0622384Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:07:10.0624073Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:07:10.3652534Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:10.4041818Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:07:10.4282136Z No Packages marked for removal 2024-06-26T07:07:10.4401429Z + install_nvidia_driver_common 2024-06-26T07:07:10.4403251Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:07:10.4403897Z + lspci 2024-06-26T07:07:10.4404204Z Before installing NVIDIA driver 2024-06-26T07:07:10.4486237Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:07:10.4487134Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:07:10.4488083Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:07:10.4488900Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:07:10.4489659Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:07:10.4490424Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:07:10.4491150Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:07:10.4491967Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:07:10.4492557Z + lsmod 2024-06-26T07:07:10.4499648Z Module Size Used by 2024-06-26T07:07:10.4500083Z nvidia_modeset 1339392 0 2024-06-26T07:07:10.4500516Z nvidia_uvm 4599808 0 2024-06-26T07:07:10.4500970Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:07:10.4501473Z drm 421888 1 nvidia 2024-06-26T07:07:10.4502747Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:07:10.4503439Z veth 16384 0 2024-06-26T07:07:10.4503942Z backlight 16384 1 nvidia_modeset 2024-06-26T07:07:10.4504425Z xt_conntrack 16384 1 2024-06-26T07:07:10.4504916Z ipt_MASQUERADE 16384 1 2024-06-26T07:07:10.4505376Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:07:10.4505864Z nf_conntrack_netlink 49152 0 2024-06-26T07:07:10.4506372Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:07:10.4506984Z xfrm_user 45056 1 2024-06-26T07:07:10.4507517Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:07:10.4508072Z iptable_nat 16384 1 2024-06-26T07:07:10.4508553Z nf_conntrack_ipv4 16384 3 2024-06-26T07:07:10.4509143Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:07:10.4509710Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:07:10.4510436Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:07:10.4511680Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:07:10.4512698Z xt_addrtype 16384 2 2024-06-26T07:07:10.4513169Z iptable_filter 16384 1 2024-06-26T07:07:10.4513685Z br_netfilter 24576 0 2024-06-26T07:07:10.4514360Z bridge 172032 1 br_netfilter 2024-06-26T07:07:10.4514914Z stp 16384 1 bridge 2024-06-26T07:07:10.4515621Z llc 16384 2 bridge,stp 2024-06-26T07:07:10.4516149Z overlay 86016 0 2024-06-26T07:07:10.4516650Z sunrpc 393216 1 2024-06-26T07:07:10.4517147Z dm_mirror 28672 0 2024-06-26T07:07:10.4517552Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:07:10.4518051Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:07:10.4518563Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:07:10.4519021Z dax 69632 1 dm_mod 2024-06-26T07:07:10.4519438Z crc32_pclmul 16384 0 2024-06-26T07:07:10.4519818Z ghash_clmulni_intel 16384 0 2024-06-26T07:07:10.4520203Z pcbc 16384 0 2024-06-26T07:07:10.4520579Z aesni_intel 188416 0 2024-06-26T07:07:10.4520961Z mousedev 24576 0 2024-06-26T07:07:10.4521352Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:07:10.4521807Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:07:10.4522275Z glue_helper 16384 1 aesni_intel 2024-06-26T07:07:10.4522881Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:07:10.4523565Z evdev 20480 3 2024-06-26T07:07:10.4523939Z psmouse 32768 0 2024-06-26T07:07:10.4524454Z button 16384 0 2024-06-26T07:07:10.4524825Z ena 139264 0 2024-06-26T07:07:10.4525196Z ptp 20480 1 ena 2024-06-26T07:07:10.4525604Z pps_core 20480 1 ptp 2024-06-26T07:07:10.4526011Z crc32c_intel 24576 0 2024-06-26T07:07:10.4526383Z autofs4 49152 2 2024-06-26T07:07:10.4526754Z + modinfo nvidia 2024-06-26T07:07:10.4527484Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:07:10.4528199Z alias: char-major-195-* 2024-06-26T07:07:10.4528585Z version: 550.54.15 2024-06-26T07:07:10.4528940Z supported: external 2024-06-26T07:07:10.4529299Z license: NVIDIA 2024-06-26T07:07:10.4529687Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:07:10.4530176Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:07:10.4530654Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:07:10.4531131Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:07:10.4531641Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:07:10.4532136Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:07:10.4532634Z depends: i2c-core,drm 2024-06-26T07:07:10.4533142Z retpoline: Y 2024-06-26T07:07:10.4533469Z name: nvidia 2024-06-26T07:07:10.4534087Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:07:10.4534758Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:07:10.4535425Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:07:10.4536026Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:07:10.4536485Z parm: NVreg_RmLogonRC:int 2024-06-26T07:07:10.4536930Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:07:10.4537399Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:07:10.4537836Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:07:10.4538289Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:07:10.4538833Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:07:10.4539407Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:07:10.4539888Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:07:10.4540336Z parm: NVreg_EnableMSI:int 2024-06-26T07:07:10.4540772Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:07:10.4541242Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:07:10.4541778Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:07:10.4542369Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:07:10.4542944Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:07:10.4543679Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:10.4544281Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:07:10.4544904Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:10.4545505Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:07:10.4546006Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:07:10.4546544Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:07:10.4547109Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:07:10.4547625Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:07:10.4548091Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:07:10.4548585Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:07:10.4549059Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:07:10.4549517Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:07:10.4550022Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:07:10.4550579Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:07:10.4551068Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:07:10.4551568Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:07:10.4552070Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:07:10.4552569Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:07:10.4553079Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:07:10.4553597Z parm: NVreg_RmMsg:charp 2024-06-26T07:07:10.4554327Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:07:10.4554864Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:07:10.4555346Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:07:10.4555808Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:07:10.4556290Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:07:10.4556776Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:07:10.4557241Z parm: rm_firmware_active:charp 2024-06-26T07:07:10.4557668Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:07:10.4558050Z ++ command -v nvidia-smi 2024-06-26T07:07:10.4558450Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:07:10.4558833Z + set +e 2024-06-26T07:07:10.4559364Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:07:12.4449935Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:07:12.4450473Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:07:12.4450978Z + '[' 0 -ne 0 ']' 2024-06-26T07:07:12.4451345Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:07:12.4452062Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:07:12.4453075Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:07:12.4454170Z + set -e 2024-06-26T07:07:12.4454586Z + '[' 1 -eq 0 ']' 2024-06-26T07:07:12.4455381Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:07:12.4456358Z + post_install_nvidia_driver_common 2024-06-26T07:07:12.4456956Z + sudo modprobe nvidia 2024-06-26T07:07:12.4552366Z + echo 'After installing NVIDIA driver' 2024-06-26T07:07:12.4552979Z + lspci 2024-06-26T07:07:12.4553410Z After installing NVIDIA driver 2024-06-26T07:07:12.4635418Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:07:12.4636183Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:07:12.4637103Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:07:12.4637919Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:07:12.4638679Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:07:12.4639430Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:07:12.4640150Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:07:12.4640933Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:07:12.4641697Z + lsmod 2024-06-26T07:07:12.4649970Z Module Size Used by 2024-06-26T07:07:12.4650391Z nvidia_modeset 1339392 0 2024-06-26T07:07:12.4650772Z nvidia_uvm 4599808 0 2024-06-26T07:07:12.4651323Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:07:12.4651960Z drm 421888 1 nvidia 2024-06-26T07:07:12.4652382Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:07:12.4652803Z veth 16384 0 2024-06-26T07:07:12.4653229Z backlight 16384 1 nvidia_modeset 2024-06-26T07:07:12.4653865Z xt_conntrack 16384 1 2024-06-26T07:07:12.4654414Z ipt_MASQUERADE 16384 1 2024-06-26T07:07:12.4654858Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:07:12.4655327Z nf_conntrack_netlink 49152 0 2024-06-26T07:07:12.4655769Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:07:12.4656235Z xfrm_user 45056 1 2024-06-26T07:07:12.4656629Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:07:12.4657056Z iptable_nat 16384 1 2024-06-26T07:07:12.4657432Z nf_conntrack_ipv4 16384 3 2024-06-26T07:07:12.4657862Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:07:12.4658458Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:07:12.4659024Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:07:12.4659968Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:07:12.4660856Z xt_addrtype 16384 2 2024-06-26T07:07:12.4661395Z iptable_filter 16384 1 2024-06-26T07:07:12.4662023Z br_netfilter 24576 0 2024-06-26T07:07:12.4662614Z bridge 172032 1 br_netfilter 2024-06-26T07:07:12.4663296Z stp 16384 1 bridge 2024-06-26T07:07:12.4663946Z llc 16384 2 bridge,stp 2024-06-26T07:07:12.4664624Z overlay 86016 0 2024-06-26T07:07:12.4665195Z sunrpc 393216 1 2024-06-26T07:07:12.4665717Z dm_mirror 28672 0 2024-06-26T07:07:12.4666274Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:07:12.4666932Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:07:12.4667580Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:07:12.4668154Z dax 69632 1 dm_mod 2024-06-26T07:07:12.4668725Z crc32_pclmul 16384 0 2024-06-26T07:07:12.4669109Z ghash_clmulni_intel 16384 0 2024-06-26T07:07:12.4669634Z pcbc 16384 0 2024-06-26T07:07:12.4670013Z aesni_intel 188416 0 2024-06-26T07:07:12.4670385Z mousedev 24576 0 2024-06-26T07:07:12.4670772Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:07:12.4671222Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:07:12.4671671Z glue_helper 16384 1 aesni_intel 2024-06-26T07:07:12.4672230Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:07:12.4672767Z evdev 20480 3 2024-06-26T07:07:12.4673136Z psmouse 32768 0 2024-06-26T07:07:12.4673512Z button 16384 0 2024-06-26T07:07:12.4674177Z ena 139264 0 2024-06-26T07:07:12.4674546Z ptp 20480 1 ena 2024-06-26T07:07:12.4674945Z pps_core 20480 1 ptp 2024-06-26T07:07:12.4675341Z crc32c_intel 24576 0 2024-06-26T07:07:12.4675716Z autofs4 49152 2 2024-06-26T07:07:12.4676071Z + modinfo nvidia 2024-06-26T07:07:12.4676746Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:07:12.4677450Z alias: char-major-195-* 2024-06-26T07:07:12.4677828Z version: 550.54.15 2024-06-26T07:07:12.4678175Z supported: external 2024-06-26T07:07:12.4678517Z license: NVIDIA 2024-06-26T07:07:12.4678896Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:07:12.4679482Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:07:12.4679943Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:07:12.4680413Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:07:12.4680914Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:07:12.4681396Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:07:12.4681883Z depends: i2c-core,drm 2024-06-26T07:07:12.4682245Z retpoline: Y 2024-06-26T07:07:12.4682553Z name: nvidia 2024-06-26T07:07:12.4683106Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:07:12.4683807Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:07:12.4684640Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:07:12.4685261Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:07:12.4685701Z parm: NVreg_RmLogonRC:int 2024-06-26T07:07:12.4686138Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:07:12.4686603Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:07:12.4687038Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:07:12.4687470Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:07:12.4688000Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:07:12.4688752Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:07:12.4689241Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:07:12.4689669Z parm: NVreg_EnableMSI:int 2024-06-26T07:07:12.4690091Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:07:12.4690555Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:07:12.4691089Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:07:12.4691664Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:07:12.4692228Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:07:12.4692840Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:12.4693434Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:07:12.4694062Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:12.4694657Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:07:12.4695151Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:07:12.4695682Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:07:12.4696233Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:07:12.4696732Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:07:12.4697295Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:07:12.4697778Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:07:12.4698247Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:07:12.4698698Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:07:12.4699204Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:07:12.4699726Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:07:12.4700202Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:07:12.4700695Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:07:12.4701194Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:07:12.4701674Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:07:12.4702172Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:07:12.4702653Z parm: NVreg_RmMsg:charp 2024-06-26T07:07:12.4703066Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:07:12.4703528Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:07:12.4704005Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:07:12.4704507Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:07:12.4705062Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:07:12.4705540Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:07:12.4706003Z parm: rm_firmware_active:charp 2024-06-26T07:07:12.4706406Z + set +e 2024-06-26T07:07:12.4706701Z + nvidia-smi 2024-06-26T07:07:14.0409231Z Wed Jun 26 07:07:14 2024 2024-06-26T07:07:14.0410546Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:14.0411532Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:07:14.0412325Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:07:14.0413107Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:07:14.0414001Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:07:14.0414744Z | | | MIG M. | 2024-06-26T07:07:14.0415286Z |=========================================+========================+======================| 2024-06-26T07:07:14.0503075Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:07:14.0503796Z | 0% 32C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:07:14.0504485Z | | | N/A | 2024-06-26T07:07:14.0505206Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:07:14.0505846Z 2024-06-26T07:07:14.0506488Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:14.0507120Z | Processes: | 2024-06-26T07:07:14.0507821Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:07:14.0508499Z | ID ID Usage | 2024-06-26T07:07:14.0509054Z |=========================================================================================| 2024-06-26T07:07:14.0509705Z | No running processes found | 2024-06-26T07:07:14.0510446Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:14.6545568Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:07:16.2167015Z NVIDIA A10G 2024-06-26T07:07:16.6515454Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:07:16.6516111Z + '[' 0 -eq 0 ']' 2024-06-26T07:07:16.6516550Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:07:16.6517307Z + set -e 2024-06-26T07:07:16.6517627Z INFO: Ignoring allowed status 0 2024-06-26T07:07:16.6518917Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:07:16.6521784Z + sudo yum install -y yum-utils 2024-06-26T07:07:16.9526225Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:18.4703307Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:07:18.4704000Z Nothing to do 2024-06-26T07:07:18.6218998Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:07:18.6219989Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:18.6221034Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:18.9489515Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:18.9752502Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:18.9753893Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:07:18.9754916Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:07:18.9863781Z + sudo yum install -y nvidia-docker2 2024-06-26T07:07:19.2907953Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:20.7620093Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:07:20.7620956Z Nothing to do 2024-06-26T07:07:20.9154483Z + sudo systemctl restart docker 2024-06-26T07:08:02.1128406Z Command completed after 1 attempt(s). 2024-06-26T07:08:02.1190306Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:08:02.1191100Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:08:02.1191816Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:08:02.1192497Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:02.1200801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:02.1201312Z env: 2024-06-26T07:08:02.1201606Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:02.1202077Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:02.1202569Z ##[endgroup] 2024-06-26T07:08:02.3409393Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:08:02.3574322Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:08:02.3674932Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:08:02.4759205Z Prepare all required actions 2024-06-26T07:08:02.4759702Z Getting action download info 2024-06-26T07:08:02.6037975Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:08:02.8159217Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:08:02.9530914Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:08:02.9531423Z with: 2024-06-26T07:08:02.9531773Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:02.9532258Z s3-bucket: gha-artifacts 2024-06-26T07:08:02.9532618Z env: 2024-06-26T07:08:02.9532902Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:02.9533381Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:02.9533907Z ##[endgroup] 2024-06-26T07:08:02.9566734Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:08:02.9567205Z with: 2024-06-26T07:08:02.9567606Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:02.9568078Z s3-bucket: gha-artifacts 2024-06-26T07:08:02.9568448Z region: us-east-1 2024-06-26T07:08:02.9568764Z env: 2024-06-26T07:08:02.9569055Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:02.9569554Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:02.9570073Z ##[endgroup] 2024-06-26T07:08:03.3724539Z (node:23307) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:08:03.3725282Z 2024-06-26T07:08:03.3725548Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:08:03.3726286Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:08:03.3727298Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:08:03.4467722Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:08:03.4469240Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:08:22.0895163Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:08:22.0901736Z Artifact download has finished successfully 2024-06-26T07:08:22.1050597Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:08:22.1051042Z unzip -o artifacts.zip 2024-06-26T07:08:22.1058656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:22.1059163Z env: 2024-06-26T07:08:22.1059437Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:22.1059905Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:22.1060393Z ##[endgroup] 2024-06-26T07:08:22.1091646Z Archive: artifacts.zip 2024-06-26T07:08:22.1093285Z creating: dist/ 2024-06-26T07:08:24.0756654Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:08:24.0757352Z creating: build/custom_test_artifacts/ 2024-06-26T07:08:24.0757947Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:08:24.0758705Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:08:24.0759551Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:24.0764834Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:24.0766213Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:24.0767556Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:24.0769011Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:24.0770424Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:24.0772072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:24.0773862Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:24.0775340Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:24.0776805Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:24.0778516Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:24.0780205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:24.0781873Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:24.0783567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:24.0785254Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:24.0786940Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:24.0788446Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:24.0789862Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:24.0822774Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:24.0861302Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:24.0863445Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:24.0906259Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:24.0908379Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:24.0910482Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:24.0912688Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:24.0915104Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:24.0917214Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:24.0919257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:24.0921459Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:24.0923467Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:24.0925490Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:24.0927263Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:24.0929062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:24.0930872Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:24.0932604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:24.0934373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:24.0978373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:24.1036748Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:24.1038531Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:24.1039996Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:24.1041266Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:24.1042607Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:24.1044021Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:08:24.1045693Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:08:24.1047316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:08:24.1048927Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:08:24.1050377Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:08:24.1051926Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:08:24.1053451Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:08:24.1055016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:08:24.1056546Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:08:24.1058079Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:08:24.1066404Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:08:24.1195567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:08:24.1197073Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:08:24.1198702Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:08:24.1200511Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:08:24.1202187Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:08:24.1203771Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:08:24.1205663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:08:24.1207340Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:08:24.1208960Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:08:24.1210626Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:08:24.1212200Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:08:24.1222382Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:08:24.1302237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:08:24.1304041Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:24.1305680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:24.1307222Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:08:24.1308546Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:08:24.1309904Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:24.1311274Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:08:24.1312513Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:08:24.1313625Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:08:24.1315111Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:08:24.1418964Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:08:24.1479737Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:08:24.1480812Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:08:24.1481779Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:08:24.1482919Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:24.1488176Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:24.1489535Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:24.1490845Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:24.1492275Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:24.1493684Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:24.1495332Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:24.1496938Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:24.1498377Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:24.1499795Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:24.1501471Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:24.1503176Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:24.1504831Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:24.1506486Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:24.1508289Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:24.1509992Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:24.1511472Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:24.1512886Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:24.1544797Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:24.1582130Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:24.1583598Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:24.1625907Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:24.1627371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:24.1628981Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:24.1630478Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:24.1631903Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:24.1633314Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:24.1634986Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:24.1636365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:24.1637727Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:24.1639011Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:24.1640258Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:24.1641479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:24.1642707Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:24.1643909Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:24.1645206Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:24.1691481Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:24.1749029Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:24.1750219Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:24.1751214Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:24.1752075Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:24.1752968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:24.1754035Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:08:24.1755107Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:08:24.1756465Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:08:24.1774513Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:08:24.1775680Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:08:24.1776813Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:08:24.1777946Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:08:24.1779081Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:08:24.1780191Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:08:24.1781289Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:08:24.1782485Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:08:24.1834986Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:08:24.1836189Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:24.1837261Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:24.1838237Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:08:24.1839148Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:08:24.1840041Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:24.1840935Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:08:24.1841797Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:08:24.1842575Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:08:24.1843369Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:08:24.1888855Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:08:24.1889606Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:08:24.1890374Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:08:24.1891263Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:24.1896228Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:24.1897222Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:24.1898233Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:24.1899308Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:24.1900366Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:24.1901546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:24.1902723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:24.1903797Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:24.1904853Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:24.1906089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:24.1907432Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:24.1908634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:24.1909847Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:24.1911083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:24.1912318Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:24.1913428Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:24.1914645Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:24.1947392Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:24.1985073Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:24.1986701Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:24.2029084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:24.2030570Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:24.2032077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:24.2033620Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:24.2035353Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:24.2036817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:24.2038290Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:24.2039747Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:24.2041159Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:24.2042509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:24.2043818Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:24.2045203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:24.2046502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:24.2047766Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:24.2049059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:24.2097211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:24.2155108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:24.2156379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:24.2157578Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:24.2158507Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:24.2159491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:24.2160503Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:08:24.2161655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:08:24.2162922Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:08:24.2164155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:08:24.2165407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:08:24.2166602Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:08:24.2167958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:08:24.2169170Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:08:24.2170357Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:08:24.2171546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:08:24.2172806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:08:24.2284820Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:08:24.2286016Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:08:24.2287216Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:08:24.2288539Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:08:24.2289804Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:08:24.2291011Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:08:24.2292251Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:08:24.2293510Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:08:24.2294766Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:08:24.2296009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:08:24.2297249Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:08:24.2307883Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:08:24.2360633Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:08:24.2361955Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:24.2363106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:24.2364152Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:08:24.2365357Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:08:24.2366333Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:24.2367301Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:08:24.2368211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:08:24.2369056Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:08:24.2369930Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:08:24.2467305Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:08:24.2507724Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:08:24.2508361Z creating: build/lib/ 2024-06-26T07:08:24.2592797Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:08:24.3030139Z inflating: build/lib/libprotobuf.a 2024-06-26T07:08:24.3039533Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:08:24.3047311Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:08:24.3054881Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:08:24.3055679Z inflating: build/lib/libclog.a 2024-06-26T07:08:24.3072723Z inflating: build/lib/libnnpack.a 2024-06-26T07:08:24.3074860Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:08:24.3137290Z inflating: build/lib/libgtest.a 2024-06-26T07:08:24.3209120Z inflating: build/lib/libbenchmark.a 2024-06-26T07:08:24.3271006Z inflating: build/lib/libasmjit.a 2024-06-26T07:08:24.3277701Z inflating: build/lib/libittnotify.a 2024-06-26T07:08:24.3304416Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:08:24.3426860Z inflating: build/lib/libgloo.a 2024-06-26T07:08:24.3447372Z inflating: build/lib/libfmt.a 2024-06-26T07:08:24.3535604Z inflating: build/lib/libc10.so 2024-06-26T07:08:24.3537042Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:08:24.3537536Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:08:24.3538510Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:08:24.3557419Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:08:24.3574610Z inflating: build/lib/libgmock.a 2024-06-26T07:08:24.3575080Z inflating: build/lib/libgtest_main.a 2024-06-26T07:08:24.3575569Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:08:24.4059107Z inflating: build/lib/libprotoc.a 2024-06-26T07:08:24.4428412Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:08:25.4143837Z inflating: build/lib/libdnnl.a 2024-06-26T07:08:25.4692914Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:08:25.4746037Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:08:25.4746947Z inflating: build/lib/libgmock_main.a 2024-06-26T07:08:25.5946955Z inflating: build/lib/libfbgemm.a 2024-06-26T07:08:25.6191567Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:08:25.6689127Z inflating: build/lib/libkineto.a 2024-06-26T07:08:25.6867637Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:08:25.6908468Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:08:25.7572657Z inflating: build/lib/libonnx.a 2024-06-26T07:08:28.0832502Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:08:28.0836129Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:08:28.0839954Z inflating: build/lib/libshm.so 2024-06-26T07:08:30.0672394Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:08:30.0672899Z inflating: build/lib/libtorch.so 2024-06-26T07:08:30.0675266Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:08:30.8608475Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:08:31.0501421Z inflating: build/lib/libtorch_python.so 2024-06-26T07:08:31.0567731Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:08:31.0587634Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:08:31.0612355Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:08:31.0645683Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:08:31.0646318Z creating: build/bin/ 2024-06-26T07:08:31.0694706Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:08:31.0744457Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:08:31.0793697Z inflating: build/bin/c10_Device_test 2024-06-26T07:08:31.0850708Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:08:31.0902458Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:08:31.0950025Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:08:31.0999376Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:08:31.1052103Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:08:31.1106015Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:08:31.1160563Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:08:31.1229467Z inflating: build/bin/c10_cow_test 2024-06-26T07:08:31.1281190Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:08:31.1328235Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:08:31.1376567Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:08:31.1425449Z inflating: build/bin/c10_Half_test 2024-06-26T07:08:31.1480352Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:08:31.1533271Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:08:31.1581698Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:08:31.1635578Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:08:31.1685552Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:08:31.1734907Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:08:31.1782190Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:08:31.1832675Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:08:31.1885944Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:08:31.1934677Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:08:31.1989464Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:08:31.2043757Z inflating: build/bin/c10_complex_test 2024-06-26T07:08:31.2094349Z inflating: build/bin/c10_exception_test 2024-06-26T07:08:31.2143197Z inflating: build/bin/c10_flags_test 2024-06-26T07:08:31.2191516Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:08:31.2350464Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:08:31.2399206Z inflating: build/bin/c10_irange_test 2024-06-26T07:08:31.2451104Z inflating: build/bin/c10_lazy_test 2024-06-26T07:08:31.2506995Z inflating: build/bin/c10_logging_test 2024-06-26T07:08:31.2578908Z inflating: build/bin/c10_optional_test 2024-06-26T07:08:31.2639918Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:08:31.2692115Z inflating: build/bin/c10_registry_test 2024-06-26T07:08:31.2837783Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:08:31.2887863Z inflating: build/bin/c10_ssize_test 2024-06-26T07:08:31.2938156Z inflating: build/bin/c10_string_util_test 2024-06-26T07:08:31.2995553Z inflating: build/bin/c10_string_view_test 2024-06-26T07:08:31.3044167Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:08:31.3091076Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:08:31.3145198Z inflating: build/bin/c10_typeid_test 2024-06-26T07:08:31.3578851Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:08:31.4012031Z inflating: build/bin/protoc 2024-06-26T07:08:31.4063416Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:08:31.4114839Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:08:31.4165953Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:08:31.4216321Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:08:31.4267276Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:08:31.4318446Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:08:31.4366107Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:08:31.4416851Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:08:31.4735230Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:08:31.5067636Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:08:31.5412944Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:08:31.5464224Z inflating: build/bin/FileStoreTest 2024-06-26T07:08:31.5518281Z inflating: build/bin/TCPStoreTest 2024-06-26T07:08:31.5569847Z inflating: build/bin/HashStoreTest 2024-06-26T07:08:31.5582843Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:08:31.5635441Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:08:31.5639037Z inflating: build/bin/torch_shm_manager 2024-06-26T07:08:31.5641615Z inflating: build/bin/example_allreduce 2024-06-26T07:08:31.5694972Z inflating: build/bin/test_dist_autograd 2024-06-26T07:08:31.5762327Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:08:31.5764532Z inflating: build/bin/parallel_benchmark 2024-06-26T07:08:31.5829680Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:08:31.5838340Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:08:31.6171534Z inflating: build/bin/test_lazy 2024-06-26T07:08:31.7316827Z inflating: build/bin/test_api 2024-06-26T07:08:31.7388195Z inflating: build/bin/Dict_test 2024-06-26T07:08:31.7438420Z inflating: build/bin/Dimname_test 2024-06-26T07:08:31.7500991Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:08:31.7556746Z inflating: build/bin/NamedTensor_test 2024-06-26T07:08:31.7613687Z inflating: build/bin/apply_utils_test 2024-06-26T07:08:31.7670701Z inflating: build/bin/atest 2024-06-26T07:08:31.7732004Z inflating: build/bin/basic 2024-06-26T07:08:31.7785040Z inflating: build/bin/broadcast_test 2024-06-26T07:08:31.7834904Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:08:31.7890777Z inflating: build/bin/cpu_generator_test 2024-06-26T07:08:31.7942201Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:08:31.8032619Z inflating: build/bin/cpu_rng_test 2024-06-26T07:08:31.8081190Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:08:31.8129943Z inflating: build/bin/dlconvertor_test 2024-06-26T07:08:31.8186255Z inflating: build/bin/extension_backend_test 2024-06-26T07:08:31.8239696Z inflating: build/bin/half_test 2024-06-26T07:08:31.8332788Z inflating: build/bin/ivalue_test 2024-06-26T07:08:31.8381159Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:08:31.8434175Z inflating: build/bin/math_kernel_test 2024-06-26T07:08:31.8486656Z inflating: build/bin/memory_format_test 2024-06-26T07:08:31.8538103Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:08:31.8589561Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:08:31.8644068Z inflating: build/bin/native_test 2024-06-26T07:08:31.8693248Z inflating: build/bin/operator_name_test 2024-06-26T07:08:31.8743147Z inflating: build/bin/operators_test 2024-06-26T07:08:31.8793953Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:08:31.8858519Z inflating: build/bin/pow_test 2024-06-26T07:08:31.8913853Z inflating: build/bin/quantized_test 2024-06-26T07:08:31.8962934Z inflating: build/bin/reduce_ops_test 2024-06-26T07:08:31.9012545Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:08:31.9067840Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:08:31.9124277Z inflating: build/bin/scalar_test 2024-06-26T07:08:31.9174674Z inflating: build/bin/StorageUtils_test 2024-06-26T07:08:31.9225497Z inflating: build/bin/stride_properties_test 2024-06-26T07:08:31.9301833Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:08:31.9354659Z inflating: build/bin/test_parallel 2024-06-26T07:08:31.9357063Z inflating: build/bin/thread_init_test 2024-06-26T07:08:31.9411127Z inflating: build/bin/type_ptr_test 2024-06-26T07:08:31.9469428Z inflating: build/bin/type_test 2024-06-26T07:08:31.9520232Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:08:31.9521694Z inflating: build/bin/verify_api_visibility 2024-06-26T07:08:31.9588592Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:08:31.9638460Z inflating: build/bin/weakref_test 2024-06-26T07:08:31.9688453Z inflating: build/bin/wrapdim_test 2024-06-26T07:08:31.9738519Z inflating: build/bin/xla_tensor_test 2024-06-26T07:08:31.9796820Z inflating: build/bin/IListRef_test 2024-06-26T07:08:31.9899640Z inflating: build/bin/List_test 2024-06-26T07:08:31.9964209Z inflating: build/bin/KernelFunction_test 2024-06-26T07:08:32.0082991Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:08:32.0176076Z inflating: build/bin/kernel_function_test 2024-06-26T07:08:32.0300431Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:08:32.0400712Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:08:32.0459848Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:08:32.0553083Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:08:32.0602387Z inflating: build/bin/CppSignature_test 2024-06-26T07:08:32.0656418Z inflating: build/bin/backend_fallback_test 2024-06-26T07:08:32.0703709Z inflating: build/bin/op_allowlist_test 2024-06-26T07:08:32.0765498Z inflating: build/bin/inline_container_test 2024-06-26T07:08:32.1057369Z inflating: build/bin/op_registration_test 2024-06-26T07:08:32.1108677Z inflating: build/bin/cuda_apply_test 2024-06-26T07:08:32.1158061Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:08:32.1215326Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:08:32.1268191Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:08:32.1335999Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:08:32.1393122Z inflating: build/bin/cuda_complex_test 2024-06-26T07:08:32.1441618Z inflating: build/bin/cuda_device_test 2024-06-26T07:08:32.1496972Z inflating: build/bin/cuda_cub_test 2024-06-26T07:08:32.1546135Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:08:32.1608967Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:08:32.1663841Z inflating: build/bin/cuda_generator_test 2024-06-26T07:08:32.1712021Z inflating: build/bin/cuda_half_test 2024-06-26T07:08:32.1761731Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:08:32.1809481Z inflating: build/bin/cuda_optional_test 2024-06-26T07:08:32.1859533Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:08:32.1911081Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:08:32.1959151Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:08:32.2017865Z inflating: build/bin/cuda_stream_test 2024-06-26T07:08:32.2065602Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:08:32.2116306Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:08:32.2130618Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:08:32.2194677Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:08:32.2249861Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:08:32.2311146Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:08:32.2371256Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:08:32.3177303Z inflating: build/bin/test_tensorexpr 2024-06-26T07:08:32.3736898Z inflating: build/bin/test_jit 2024-06-26T07:08:32.3737331Z creating: .additional_ci_files/ 2024-06-26T07:08:32.3790784Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:08:32.4002160Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:08:32.4028833Z ##[group]Run rm artifacts.zip 2024-06-26T07:08:32.4029227Z rm artifacts.zip 2024-06-26T07:08:32.4038612Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:32.4039116Z env: 2024-06-26T07:08:32.4039417Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:32.4039868Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:32.4040361Z ##[endgroup] 2024-06-26T07:08:32.4710289Z ##[group]Run df -H 2024-06-26T07:08:32.4710620Z df -H 2024-06-26T07:08:32.4718486Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:32.4718988Z env: 2024-06-26T07:08:32.4719273Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:32.4719736Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:32.4720217Z ##[endgroup] 2024-06-26T07:08:32.4746851Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:08:32.4747430Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:08:32.4747905Z tmpfs 34G 25k 34G 1% /dev/shm 2024-06-26T07:08:32.4748631Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:08:32.4749125Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:08:32.4749697Z /dev/nvme0n1p1 162G 49G 113G 31% / 2024-06-26T07:08:32.4787781Z Prepare all required actions 2024-06-26T07:08:32.4788210Z Getting action download info 2024-06-26T07:08:32.6070352Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:08:32.6070818Z with: 2024-06-26T07:08:32.6071075Z env: 2024-06-26T07:08:32.6071353Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:32.6071813Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:32.6072290Z ##[endgroup] 2024-06-26T07:08:32.6116336Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:08:32.6116782Z with: 2024-06-26T07:08:32.6117043Z name: td_results 2024-06-26T07:08:32.6117359Z s3-bucket: gha-artifacts 2024-06-26T07:08:32.6117705Z region: us-east-1 2024-06-26T07:08:32.6118005Z env: 2024-06-26T07:08:32.6118268Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:32.6118732Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:32.6119210Z ##[endgroup] 2024-06-26T07:08:33.0195147Z (node:23335) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:08:33.0195873Z 2024-06-26T07:08:33.0196147Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:08:33.0196893Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:08:33.0197766Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:08:33.0966701Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:08:33.0973259Z Artifact download has finished successfully 2024-06-26T07:08:33.1098487Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:08:33.1098982Z mkdir -p .additional_ci_files 2024-06-26T07:08:33.1099551Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:08:33.1107487Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:33.1107997Z env: 2024-06-26T07:08:33.1108281Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.1108762Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.1109244Z ##[endgroup] 2024-06-26T07:08:33.1158250Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:08:33.1169108Z ##[error]Process completed with exit code 1. 2024-06-26T07:08:33.1201889Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:08:33.1202377Z .github/scripts/parse_ref.py 2024-06-26T07:08:33.1209635Z shell: /usr/bin/bash -e {0} 2024-06-26T07:08:33.1209982Z env: 2024-06-26T07:08:33.1210264Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.1210725Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.1211212Z ##[endgroup] 2024-06-26T07:08:33.1440382Z Prepare all required actions 2024-06-26T07:08:33.1477574Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:08:33.1478170Z with: 2024-06-26T07:08:33.1478723Z github-token: *** 2024-06-26T07:08:33.1479036Z env: 2024-06-26T07:08:33.1479313Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.1479776Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.1480263Z ##[endgroup] 2024-06-26T07:08:33.1495308Z ##[group]Run set -eux 2024-06-26T07:08:33.1495648Z set -eux 2024-06-26T07:08:33.1496242Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:08:33.1504054Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:33.1504551Z env: 2024-06-26T07:08:33.1504832Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.1505290Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.1505935Z GITHUB_TOKEN: *** 2024-06-26T07:08:33.1506243Z ##[endgroup] 2024-06-26T07:08:33.1529367Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0467b1cafd1897be0 2024-06-26T07:08:34.6116481Z setting job-id=26688912257 2024-06-26T07:08:34.6117400Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:34.6313229Z Prepare all required actions 2024-06-26T07:08:34.6313664Z Getting action download info 2024-06-26T07:08:34.7432936Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:08:34.7433401Z with: 2024-06-26T07:08:34.7434013Z github-token: *** 2024-06-26T07:08:34.7442220Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:08:34.7451117Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:34.7451855Z env: 2024-06-26T07:08:34.7452135Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:34.7452591Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:34.7453065Z ##[endgroup] 2024-06-26T07:08:34.7493568Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:08:34.7494129Z with: 2024-06-26T07:08:34.7494400Z shell: bash 2024-06-26T07:08:34.7494852Z timeout_minutes: 10 2024-06-26T07:08:34.7495186Z max_attempts: 5 2024-06-26T07:08:34.7495499Z retry_wait_seconds: 30 2024-06-26T07:08:34.7496632Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:08:34.7497808Z polling_interval_seconds: 1 2024-06-26T07:08:34.7498187Z warning_on_retry: true 2024-06-26T07:08:34.7498533Z continue_on_error: false 2024-06-26T07:08:34.7498877Z env: 2024-06-26T07:08:34.7499157Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:34.7499617Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:34.7500264Z GITHUB_TOKEN: *** 2024-06-26T07:08:34.7500588Z ##[endgroup] 2024-06-26T07:08:34.8001413Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:08:35.0108930Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:08:35.0264568Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:08:35.0404816Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:08:35.0412855Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:08:35.0421202Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:08:35.0432963Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:08:35.0625944Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:08:35.7996603Z Command completed after 1 attempt(s). 2024-06-26T07:08:35.8037783Z ##[group]Run set -x 2024-06-26T07:08:35.8038119Z set -x 2024-06-26T07:08:35.8038415Z  2024-06-26T07:08:35.8038984Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:08:35.8039658Z # in runner workspace 2024-06-26T07:08:35.8040189Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:08:35.8048079Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:35.8048577Z env: 2024-06-26T07:08:35.8048845Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:35.8049300Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:35.8049783Z ##[endgroup] 2024-06-26T07:08:35.8070192Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:08:35.8278933Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:08:35.8279493Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:08:35.8279980Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:08:35.8280377Z  2024-06-26T07:08:35.8280946Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:08:35.8281647Z # in runner workspace 2024-06-26T07:08:35.8282241Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:08:35.8282889Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:08:35.8283358Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:08:35.8292019Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:08:35.8300781Z  --selected-test-configs "" \ 2024-06-26T07:08:35.8301252Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:08:35.8301695Z  --tag "${TAG}" \ 2024-06-26T07:08:35.8302105Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:08:35.8302564Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:08:35.8303010Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:08:35.8310531Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:35.8311053Z env: 2024-06-26T07:08:35.8311334Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:35.8311808Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:35.8312498Z GITHUB_TOKEN: *** 2024-06-26T07:08:35.8313184Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:35.8314227Z PR_NUMBER: 2024-06-26T07:08:35.8314545Z TAG: ciflow/inductor/129470 2024-06-26T07:08:35.8314921Z EVENT_NAME: push 2024-06-26T07:08:35.8315236Z SCHEDULE: 2024-06-26T07:08:35.8315523Z HEAD_BRANCH: 2024-06-26T07:08:35.8315825Z ##[endgroup] 2024-06-26T07:08:35.8334707Z Workflow: inductor 2024-06-26T07:08:35.8335603Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:36.0555075Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:08:36.2076089Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:08:36.2076585Z echo "Filtered matrix:" 2024-06-26T07:08:36.2085045Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:08:36.2093854Z  2024-06-26T07:08:36.2094136Z echo 2024-06-26T07:08:36.2094523Z echo "Is the current job unstable? False" 2024-06-26T07:08:36.2094977Z  2024-06-26T07:08:36.2095255Z echo 2024-06-26T07:08:36.2095618Z echo "Is keep-going label set? False" 2024-06-26T07:08:36.2096059Z  2024-06-26T07:08:36.2096322Z echo 2024-06-26T07:08:36.2096638Z echo "Renabled issues? " 2024-06-26T07:08:36.2104424Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:36.2104916Z env: 2024-06-26T07:08:36.2105197Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.2105656Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.2106137Z ##[endgroup] 2024-06-26T07:08:36.2125803Z Filtered matrix: 2024-06-26T07:08:36.2135856Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:08:36.2144259Z 2024-06-26T07:08:36.2144417Z Is the current job unstable? False 2024-06-26T07:08:36.2144702Z 2024-06-26T07:08:36.2144963Z Is keep-going label set? False 2024-06-26T07:08:36.2145230Z 2024-06-26T07:08:36.2145354Z Renabled issues? 2024-06-26T07:08:36.2204265Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:36.2205124Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:36.2212860Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:36.2213357Z env: 2024-06-26T07:08:36.2213639Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.2214098Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.2214587Z JOB_TIMEOUT: 240 2024-06-26T07:08:36.2214887Z ##[endgroup] 2024-06-26T07:08:36.2283772Z ##[group]Run set -x 2024-06-26T07:08:36.2284172Z set -x 2024-06-26T07:08:36.2284636Z  2024-06-26T07:08:36.2284998Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:08:36.2285552Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:08:36.2286127Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:08:36.2286644Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:08:36.2287068Z else 2024-06-26T07:08:36.2287427Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:08:36.2287857Z fi 2024-06-26T07:08:36.2288144Z  2024-06-26T07:08:36.2288621Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:08:36.2289390Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:08:36.2290079Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:08:36.2290669Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:08:36.2291134Z container_name=$(docker run \ 2024-06-26T07:08:36.2291559Z  ${GPU_FLAG:-} \ 2024-06-26T07:08:36.2291939Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:08:36.2292347Z  -e PR_NUMBER \ 2024-06-26T07:08:36.2292710Z  -e GITHUB_ACTIONS \ 2024-06-26T07:08:36.2293113Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:08:36.2293528Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:08:36.2293922Z  -e GITHUB_JOB \ 2024-06-26T07:08:36.2294292Z  -e GITHUB_RUN_ID \ 2024-06-26T07:08:36.2294684Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:08:36.2295099Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:08:36.2295497Z  -e JOB_ID \ 2024-06-26T07:08:36.2295843Z  -e JOB_NAME \ 2024-06-26T07:08:36.2296196Z  -e BASE_SHA \ 2024-06-26T07:08:36.2296545Z  -e BRANCH \ 2024-06-26T07:08:36.2296877Z  -e SHA1 \ 2024-06-26T07:08:36.2297232Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:08:36.2297697Z  -e IN_WHEEL_TEST \ 2024-06-26T07:08:36.2298081Z  -e SHARD_NUMBER \ 2024-06-26T07:08:36.2298451Z  -e TEST_CONFIG \ 2024-06-26T07:08:36.2298831Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:08:36.2299231Z  -e REENABLED_ISSUES \ 2024-06-26T07:08:36.2299651Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:08:36.2300078Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:08:36.2300488Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:08:36.2300880Z  -e NO_TD \ 2024-06-26T07:08:36.2301222Z  -e TD_DISTRIBUTED \ 2024-06-26T07:08:36.2301609Z  -e PR_LABELS \ 2024-06-26T07:08:36.2302030Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:08:36.2302494Z  -e SCCACHE_BUCKET \ 2024-06-26T07:08:36.2302892Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:08:36.2303310Z  -e XLA_CUDA \ 2024-06-26T07:08:36.2303725Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:08:36.2304246Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:08:36.2304773Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:08:36.2305298Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:08:36.2305779Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:08:36.2306213Z  -e DASHBOARD_TAG \ 2024-06-26T07:08:36.2306697Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:08:36.2307374Z  --security-opt seccomp=unconfined \ 2024-06-26T07:08:36.2307849Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:08:36.2308250Z  --ipc=host \ 2024-06-26T07:08:36.2308611Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:08:36.2309018Z  --tty \ 2024-06-26T07:08:36.2309342Z  --detach \ 2024-06-26T07:08:36.2309710Z  --name="${container_name}" \ 2024-06-26T07:08:36.2310131Z  --user jenkins \ 2024-06-26T07:08:36.2310689Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:08:36.2311269Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:08:36.2311721Z  "${DOCKER_IMAGE}" 2024-06-26T07:08:36.2312069Z ) 2024-06-26T07:08:36.2312482Z # Propagate download.pytorch.org IP to container 2024-06-26T07:08:36.2313413Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:08:36.2314653Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:08:36.2315592Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:08:36.2322969Z shell: /usr/bin/bash -e {0} 2024-06-26T07:08:36.2323335Z env: 2024-06-26T07:08:36.2323625Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.2324092Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.2324863Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:36.2325377Z PR_NUMBER: 2024-06-26T07:08:36.2325710Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:08:36.2326132Z GITHUB_WORKFLOW: inductor 2024-06-26T07:08:36.2326495Z GITHUB_JOB: test 2024-06-26T07:08:36.2326822Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:08:36.2327192Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:08:36.2327547Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:08:36.2327887Z JOB_ID: 26688912257 2024-06-26T07:08:36.2328574Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:36.2329319Z BRANCH: 2024-06-26T07:08:36.2329655Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:36.2330177Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:36.2330676Z TEST_CONFIG: aot_inductor_huggingface 2024-06-26T07:08:36.2331097Z SHARD_NUMBER: 1 2024-06-26T07:08:36.2331405Z NUM_TEST_SHARDS: 1 2024-06-26T07:08:36.2331738Z REENABLED_ISSUES: 2024-06-26T07:08:36.2332092Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:08:36.2332485Z VERBOSE_TEST_LOGS: False 2024-06-26T07:08:36.2332848Z NO_TEST_TIMEOUT: False 2024-06-26T07:08:36.2333193Z NO_TD: False 2024-06-26T07:08:36.2333499Z TD_DISTRIBUTED: False 2024-06-26T07:08:36.2333917Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:08:36.2334418Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:08:36.2334810Z SHM_SIZE: 2g 2024-06-26T07:08:36.2335904Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:36.2337076Z XLA_CUDA: 2024-06-26T07:08:36.2337596Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:08:36.2338244Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:08:36.2338690Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:08:36.2339098Z DASHBOARD_TAG: 2024-06-26T07:08:36.2339624Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:08:36.2340005Z ##[endgroup] 2024-06-26T07:08:36.2359924Z + [[ aot_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:08:36.2360672Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:08:36.2361196Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:08:36.2366863Z +++ nproc --ignore=2 2024-06-26T07:08:36.2379181Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:47.4586220Z + container_name=6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:08:47.4587121Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:08:47.4589503Z + docker exec -i 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:08:47.5152503Z + echo DOCKER_CONTAINER_ID=6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:08:47.5154493Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:08:47.5156524Z + docker exec -t 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:08:47.8888795Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:08:48.2005754Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:08:48.2010853Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:08:48.2013405Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:08:48.2016779Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:08:48.2019338Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:08:48.2022655Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:08:48.2049252Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:08:48.2109867Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:08:48.2542461Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:08:48.2701830Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:08:49.5671878Z Installing collected packages: torch 2024-06-26T07:08:58.2835098Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:08:58.2836775Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:08:58.2837465Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:08:58.3624393Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:08:58.3627909Z + source .ci/pytorch/common.sh 2024-06-26T07:08:58.3629565Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:08:58.3635071Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:08:58.3635995Z +++ declare -f -t trap_add 2024-06-26T07:08:58.3641305Z ++ set -ex 2024-06-26T07:08:58.3641974Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:08:58.3642502Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:08:58.3643022Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:08:58.3645539Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:08:58.3655095Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:08:58.3655617Z + trap_add cleanup_workspace EXIT 2024-06-26T07:08:58.3656044Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:08:58.3656407Z + shift 2024-06-26T07:08:58.3656691Z + for trap_add_name in "$@" 2024-06-26T07:08:58.3661814Z +++ trap -p EXIT 2024-06-26T07:08:58.3663331Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:08:58.3663733Z +++ extract_trap_cmd 2024-06-26T07:08:58.3664123Z +++ printf '%s\n' '' 2024-06-26T07:08:58.3664508Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:08:58.3666109Z + trap -- ' 2024-06-26T07:08:58.3666460Z cleanup_workspace' EXIT 2024-06-26T07:08:58.3666921Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:08:58.7805807Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:08:58.7818818Z + echo 'Environment variables:' 2024-06-26T07:08:58.7819383Z Environment variables: 2024-06-26T07:08:58.7819805Z + env 2024-06-26T07:08:58.7824596Z INSTALLED_DB=yes 2024-06-26T07:08:58.7825181Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:08:58.7825680Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:08:58.7826241Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.7827100Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:08:58.7827928Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:08:58.7828644Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:08:58.7829387Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.7830259Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:58.7830987Z HOSTNAME=6b4591b4f61d 2024-06-26T07:08:58.7832271Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.7833440Z GITHUB_ACTION=__self 2024-06-26T07:08:58.7834197Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:08:58.7839783Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:08:58.7846298Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:08:58.7847070Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:08:58.7847562Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:08:58.7848089Z TEST_CONFIG=aot_inductor_huggingface 2024-06-26T07:08:58.7848729Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:08:58.7849446Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:08:58.7850236Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.7851165Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:08:58.7851795Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:08:58.7852452Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.7853227Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:08:58.7853957Z GITHUB_REF_TYPE=tag 2024-06-26T07:08:58.7854450Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:08:58.7855056Z NCCL_VERSION=2.17.1-1 2024-06-26T07:08:58.7855614Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.7856440Z XLA_CUDA= 2024-06-26T07:08:58.7857188Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:08:58.7859126Z *** 2024-06-26T07:08:58.7859449Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:08:58.7859864Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:08:58.7860242Z GITHUB_ACTIONS=true 2024-06-26T07:08:58.7860573Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:58.7861099Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:08:58.7861645Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:08:58.7862117Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.7862616Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:08:58.7863093Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.7863893Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.7864600Z UCC_HOME=/usr 2024-06-26T07:08:58.7864941Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:08:58.7865344Z VERBOSE_TEST_LOGS=False 2024-06-26T07:08:58.7865690Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:08:58.7866144Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:08:58.7866649Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.7867128Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:08:58.7867512Z SHARD_NUMBER=1 2024-06-26T07:08:58.7867824Z GITHUB_REF_PROTECTED=false 2024-06-26T07:08:58.7868179Z HOME=/var/lib/jenkins 2024-06-26T07:08:58.7868547Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:08:58.7868995Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:08:58.7869481Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:08:58.7869974Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:08:58.7870351Z CUDA_VERSION=12.1.1 2024-06-26T07:08:58.7870768Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:08:58.7871211Z NUM_TEST_SHARDS=1 2024-06-26T07:08:58.7871513Z UCX_HOME=/usr 2024-06-26T07:08:58.7872022Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:08:58.7873092Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.7874750Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:58.7875987Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.7877256Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:08:58.7877916Z GITHUB_EVENT_NAME=push 2024-06-26T07:08:58.7878247Z DASHBOARD_TAG= 2024-06-26T07:08:58.7878559Z GITHUB_RUN_ID=9673646808 2024-06-26T07:08:58.7879027Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:08:58.7879567Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:08:58.7880578Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.7881507Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.7881925Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:08:58.7882305Z PR_NUMBER= 2024-06-26T07:08:58.7882598Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:08:58.7882950Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:08:58.7883393Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:08:58.7883874Z TERM=xterm 2024-06-26T07:08:58.7884369Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:08:58.7884794Z INSTALLED_VISION=yes 2024-06-26T07:08:58.7885098Z BRANCH= 2024-06-26T07:08:58.7885528Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:08:58.7885934Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:08:58.7886356Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:08:58.7887102Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:08:58.7887863Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:08:58.7888353Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:08:58.7888817Z REENABLED_ISSUES= 2024-06-26T07:08:58.7889195Z SHLVL=1 2024-06-26T07:08:58.7889468Z MAX_JOBS=14 2024-06-26T07:08:58.7889783Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:08:58.7890125Z NVARCH=x86_64 2024-06-26T07:08:58.7890422Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:08:58.7890871Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.7891407Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:08:58.7891885Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:08:58.7892557Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:08:58.7893154Z GITHUB_JOB=test 2024-06-26T07:08:58.7893548Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:08:58.7894110Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:08:58.7894624Z NO_TEST_TIMEOUT=False 2024-06-26T07:08:58.7894954Z TD_DISTRIBUTED=False 2024-06-26T07:08:58.7895320Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:08:58.7895768Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:08:58.7896198Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:08:58.7896567Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:08:58.7896907Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:08:58.7897263Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:08:58.7898306Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:08:58.7899381Z GITHUB_BASE_REF= 2024-06-26T07:08:58.7899709Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:08:58.7900085Z CI=true 2024-06-26T07:08:58.7900401Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.7900806Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:08:58.7901183Z JOB_ID=26688912257 2024-06-26T07:08:58.7901499Z INSTALLED_PROTOBUF=yes 2024-06-26T07:08:58.7901828Z GITHUB_HEAD_REF= 2024-06-26T07:08:58.7902121Z GITHUB_ACTION_REF= 2024-06-26T07:08:58.7902554Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:08:58.7903031Z GITHUB_WORKFLOW=inductor 2024-06-26T07:08:58.7903394Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:08:58.7904284Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.7905098Z NO_TD=False 2024-06-26T07:08:58.7905399Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:08:58.7905761Z _=/usr/bin/env 2024-06-26T07:08:58.7906238Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:08:58.7979823Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:08:58.7981079Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:08:58.7981997Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:08:58.7982981Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:08:58.7983644Z + BUILD_DIR=build 2024-06-26T07:08:58.7984094Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:08:58.7984559Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:08:58.7984901Z + SHARD_NUMBER=1 2024-06-26T07:08:58.7985206Z + NUM_TEST_SHARDS=1 2024-06-26T07:08:58.7985524Z + export VALGRIND=ON 2024-06-26T07:08:58.7985838Z + VALGRIND=ON 2024-06-26T07:08:58.7986307Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:08:58.7986802Z + [[ 0 == \1 ]] 2024-06-26T07:08:58.7987096Z + [[ False == \1 ]] 2024-06-26T07:08:58.7987555Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:08:58.7988090Z ++ realpath build/custom_test_artifacts 2024-06-26T07:08:58.7991120Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:08:58.7991923Z + [[ -n '' ]] 2024-06-26T07:08:58.7992353Z + echo 'Environment variables' 2024-06-26T07:08:58.7992745Z Environment variables 2024-06-26T07:08:58.7993121Z + env 2024-06-26T07:08:58.7996401Z INSTALLED_DB=yes 2024-06-26T07:08:58.7996922Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:08:58.7997490Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:08:58.7998027Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.7999035Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:08:58.7999875Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:08:58.8000502Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:08:58.8001018Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.8001585Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:58.8002073Z HOSTNAME=6b4591b4f61d 2024-06-26T07:08:58.8002897Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.8003717Z GITHUB_ACTION=__self 2024-06-26T07:08:58.8004112Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:08:58.8008105Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:08:58.8012017Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:08:58.8012521Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:08:58.8012870Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:08:58.8013231Z TEST_CONFIG=aot_inductor_huggingface 2024-06-26T07:08:58.8013671Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:08:58.8014177Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:08:58.8014636Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.8015076Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:08:58.8015496Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:08:58.8015934Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.8016432Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:08:58.8016912Z GITHUB_REF_TYPE=tag 2024-06-26T07:08:58.8017230Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:08:58.8017602Z NCCL_VERSION=2.17.1-1 2024-06-26T07:08:58.8017990Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8018430Z XLA_CUDA= 2024-06-26T07:08:58.8018912Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:08:58.8019313Z *** 2024-06-26T07:08:58.8019599Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:08:58.8020001Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:08:58.8020366Z GITHUB_ACTIONS=true 2024-06-26T07:08:58.8020701Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:58.8021192Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:08:58.8021722Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:08:58.8022191Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8022699Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:08:58.8023183Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8023988Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.8024704Z UCC_HOME=/usr 2024-06-26T07:08:58.8025041Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:08:58.8025446Z VERBOSE_TEST_LOGS=False 2024-06-26T07:08:58.8025792Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:08:58.8026401Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:08:58.8026906Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.8027374Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:08:58.8027748Z SHARD_NUMBER=1 2024-06-26T07:08:58.8028058Z GITHUB_REF_PROTECTED=false 2024-06-26T07:08:58.8028410Z HOME=/var/lib/jenkins 2024-06-26T07:08:58.8028779Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:08:58.8031817Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:08:58.8032398Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:08:58.8032890Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:08:58.8033263Z CUDA_VERSION=12.1.1 2024-06-26T07:08:58.8033694Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:08:58.8034493Z NUM_TEST_SHARDS=1 2024-06-26T07:08:58.8034800Z UCX_HOME=/usr 2024-06-26T07:08:58.8035317Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:08:58.8036375Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.8037636Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:58.8038861Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.8039966Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:08:58.8059350Z GITHUB_EVENT_NAME=push 2024-06-26T07:08:58.8059743Z DASHBOARD_TAG= 2024-06-26T07:08:58.8060067Z GITHUB_RUN_ID=9673646808 2024-06-26T07:08:58.8060549Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:08:58.8061083Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:08:58.8062066Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.8062974Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.8063379Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:08:58.8063749Z PR_NUMBER= 2024-06-26T07:08:58.8064029Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:08:58.8064349Z VALGRIND=ON 2024-06-26T07:08:58.8064644Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:08:58.8065086Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:08:58.8065552Z TERM=xterm 2024-06-26T07:08:58.8065879Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:08:58.8066300Z INSTALLED_VISION=yes 2024-06-26T07:08:58.8066607Z BRANCH= 2024-06-26T07:08:58.8066892Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:08:58.8067292Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:08:58.8067744Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:08:58.8068485Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:08:58.8069240Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:08:58.8069722Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:08:58.8070182Z REENABLED_ISSUES= 2024-06-26T07:08:58.8070476Z SHLVL=1 2024-06-26T07:08:58.8070728Z MAX_JOBS=14 2024-06-26T07:08:58.8071036Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:08:58.8071393Z NVARCH=x86_64 2024-06-26T07:08:58.8071683Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:08:58.8072142Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8072683Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:08:58.8073162Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:08:58.8074204Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:08:58.8074785Z GITHUB_JOB=test 2024-06-26T07:08:58.8075190Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:08:58.8075744Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:08:58.8076255Z NO_TEST_TIMEOUT=False 2024-06-26T07:08:58.8076590Z TD_DISTRIBUTED=False 2024-06-26T07:08:58.8076970Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:08:58.8077407Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:08:58.8077827Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:08:58.8078348Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:08:58.8078697Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:08:58.8079052Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:08:58.8080103Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:08:58.8081176Z GITHUB_BASE_REF= 2024-06-26T07:08:58.8081501Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:08:58.8081874Z CI=true 2024-06-26T07:08:58.8082260Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.8082685Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:08:58.8083055Z JOB_ID=26688912257 2024-06-26T07:08:58.8083371Z INSTALLED_PROTOBUF=yes 2024-06-26T07:08:58.8083701Z GITHUB_HEAD_REF= 2024-06-26T07:08:58.8084008Z GITHUB_ACTION_REF= 2024-06-26T07:08:58.8084570Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:08:58.8085046Z GITHUB_WORKFLOW=inductor 2024-06-26T07:08:58.8085408Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:08:58.8086307Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ce285370-a076-4433-a340-027ef398a3d4 2024-06-26T07:08:58.8087129Z NO_TD=False 2024-06-26T07:08:58.8087434Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:08:58.8087800Z _=/usr/bin/env 2024-06-26T07:08:58.8088128Z + echo 'Testing pytorch' 2024-06-26T07:08:58.8088474Z Testing pytorch 2024-06-26T07:08:58.8088797Z + export LANG=C.UTF-8 2024-06-26T07:08:58.8089132Z + LANG=C.UTF-8 2024-06-26T07:08:58.8089422Z + PR_NUMBER= 2024-06-26T07:08:58.8089785Z + [[ aot_inductor_huggingface == \d\e\f\a\u\l\t ]] 2024-06-26T07:08:58.8090349Z + [[ aot_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:08:58.8090879Z + [[ aot_inductor_huggingface == \s\l\o\w ]] 2024-06-26T07:08:58.8091534Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:08:58.8092227Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:08:58.8092781Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:08:58.8093259Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:08:58.8093739Z + [[ aot_inductor_huggingface == *crossref* ]] 2024-06-26T07:08:58.8094328Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:08:58.8094961Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:08:58.8095603Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:08:58.8096164Z + pip_install --user ninja==1.10.2 2024-06-26T07:08:58.8096704Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:08:59.1637038Z Collecting ninja==1.10.2 2024-06-26T07:08:59.1814246Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:08:59.1955095Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:09:00.4776889Z Installing collected packages: ninja 2024-06-26T07:09:00.4845034Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:09:00.4846330Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:09:00.4883442Z Successfully installed ninja-1.10.2 2024-06-26T07:09:00.5714068Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:00.5716268Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:00.5717701Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:09:00.5718214Z + install_tlparse 2024-06-26T07:09:00.5718589Z + pip_install --user tlparse==0.3.7 2024-06-26T07:09:00.5719143Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:09:00.9229647Z Collecting tlparse==0.3.7 2024-06-26T07:09:00.9394981Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:09:00.9506815Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:09:02.2802174Z Installing collected packages: tlparse 2024-06-26T07:09:02.3159917Z Successfully installed tlparse-0.3.7 2024-06-26T07:09:02.3992851Z ++ python -m site --user-base 2024-06-26T07:09:02.4150356Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:02.4154560Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:09:02.4155317Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:09:02.4155987Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:02.4156946Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:09:02.4158085Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:09:02.4158789Z + cd test 2024-06-26T07:09:02.4159315Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:09:03.8980202Z + [[ aot_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:09:03.8981201Z + [[ aot_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:09:03.8982242Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:03.8982937Z + pushd test 2024-06-26T07:09:03.8983373Z ~/workspace/test ~/workspace 2024-06-26T07:09:03.8984137Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:09:05.3804454Z + CUDA_VERSION=12.1 2024-06-26T07:09:05.3805105Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:09:05.3805457Z + ISCUDA124= 2024-06-26T07:09:05.3805739Z + popd 2024-06-26T07:09:05.3806031Z ~/workspace 2024-06-26T07:09:05.3809043Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:09:05.3809928Z + [[ aot_inductor_huggingface == *dynamo_eager* ]] 2024-06-26T07:09:05.3810612Z + [[ aot_inductor_huggingface == *aot_eager* ]] 2024-06-26T07:09:05.3811187Z + [[ aot_inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:09:05.3811849Z + DYNAMO_BENCHMARK_FLAGS+=(--export-aot-inductor) 2024-06-26T07:09:05.3812377Z + [[ aot_inductor_huggingface == *dynamic* ]] 2024-06-26T07:09:05.3812956Z + [[ aot_inductor_huggingface == *cpu_inductor* ]] 2024-06-26T07:09:05.3813550Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:09:05.3837392Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:09:05.3838155Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:09:05.3840343Z + cd test 2024-06-26T07:09:05.3841195Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:09:06.7349233Z PyTorch built with: 2024-06-26T07:09:06.7349899Z - GCC 9.4 2024-06-26T07:09:06.7350396Z - C++ Version: 201703 2024-06-26T07:09:06.7351483Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:09:06.7352577Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:09:06.7353247Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:09:06.7354013Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:09:06.7354509Z - NNPACK is enabled 2024-06-26T07:09:06.7354894Z - CPU capability usage: AVX2 2024-06-26T07:09:06.7355298Z - CUDA Runtime 12.1 2024-06-26T07:09:06.7355817Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:09:06.7356421Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:09:06.7356857Z - Magma 2.6.1 2024-06-26T07:09:06.7364289Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:09:06.7371137Z 2024-06-26T07:09:07.0109198Z + cd test 2024-06-26T07:09:07.0109961Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:09:08.2258487Z ATen/Parallel: 2024-06-26T07:09:08.2259006Z at::get_num_threads() : 8 2024-06-26T07:09:08.2259422Z at::get_num_interop_threads() : 16 2024-06-26T07:09:08.2259862Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:09:08.2260288Z omp_get_max_threads() : 8 2024-06-26T07:09:08.2261265Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:09:08.2262081Z mkl_get_max_threads() : 8 2024-06-26T07:09:08.2262699Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:09:08.2263344Z std::thread::hardware_concurrency() : 16 2024-06-26T07:09:08.2263781Z Environment variables: 2024-06-26T07:09:08.2264136Z OMP_NUM_THREADS : [not set] 2024-06-26T07:09:08.2264521Z MKL_NUM_THREADS : [not set] 2024-06-26T07:09:08.2264906Z ATen parallel backend: OpenMP 2024-06-26T07:09:08.2265165Z 2024-06-26T07:09:08.4718965Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:09:08.4719779Z + [[ aot_inductor_huggingface == *backward* ]] 2024-06-26T07:09:08.4720448Z + [[ aot_inductor_huggingface == *xla* ]] 2024-06-26T07:09:08.4721115Z + [[ aot_inductor_huggingface == *executorch* ]] 2024-06-26T07:09:08.4721873Z + [[ aot_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:09:08.4722651Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:09:08.4723220Z + [[ aot_inductor_huggingface == distributed ]] 2024-06-26T07:09:08.4723786Z + [[ aot_inductor_huggingface == *inductor_distributed* ]] 2024-06-26T07:09:08.4724580Z + [[ aot_inductor_huggingface == *inductor-halide* ]] 2024-06-26T07:09:08.4725234Z + [[ aot_inductor_huggingface == *inductor-micro-benchmark* ]] 2024-06-26T07:09:08.4725813Z + [[ aot_inductor_huggingface == *huggingface* ]] 2024-06-26T07:09:08.4726291Z + install_torchvision 2024-06-26T07:09:08.4726635Z + local orig_preload 2024-06-26T07:09:08.4726963Z + local commit 2024-06-26T07:09:08.4727279Z ++ get_pinned_commit vision 2024-06-26T07:09:08.4727682Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:09:08.4732882Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.4733512Z + orig_preload= 2024-06-26T07:09:08.4733968Z + '[' -n '' ']' 2024-06-26T07:09:08.4735057Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.4736503Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.7824253Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.7826854Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-ysoam6kz 2024-06-26T07:09:08.7839005Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-ysoam6kz 2024-06-26T07:09:10.1060779Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:09:10.1078468Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:11.3226697Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:11.5611142Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:13.8811811Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:09:13.8859938Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:09:13.8862210Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:09:13.8867457Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:09:13.9067003Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:09:13.9073469Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:09:13.9076035Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:09:13.9078668Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:09:13.9081564Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:09:13.9083927Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:09:13.9497367Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:09:13.9647967Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:09:13.9719776Z Building wheels for collected packages: torchvision 2024-06-26T07:10:28.2444861Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T07:10:28.2470930Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112894 sha256=cc3ad7eb569fd9df682b734edec089435335d6eeb50aad3ff9cfbe5867037f2c 2024-06-26T07:10:28.2472761Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:10:28.2507255Z Successfully built torchvision 2024-06-26T07:10:29.4394226Z Installing collected packages: torchvision 2024-06-26T07:10:29.8176483Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:10:29.9271983Z + '[' -n '' ']' 2024-06-26T07:10:29.9272398Z + id=0 2024-06-26T07:10:29.9272807Z + test_dynamo_benchmark huggingface 0 2024-06-26T07:10:29.9274761Z ++ pwd 2024-06-26T07:10:29.9276605Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:29.9277366Z + local suite=huggingface 2024-06-26T07:10:29.9277734Z + shift 2024-06-26T07:10:29.9278012Z + local shard_id=0 2024-06-26T07:10:29.9278586Z + shift 2024-06-26T07:10:29.9279043Z + [[ aot_inductor_huggingface == *perf_compare* ]] 2024-06-26T07:10:29.9279765Z + [[ aot_inductor_huggingface == *perf* ]] 2024-06-26T07:10:29.9282308Z + [[ aot_inductor_huggingface == *cpu_inductor* ]] 2024-06-26T07:10:29.9282982Z + [[ aot_inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:10:29.9283751Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-06-26T07:10:29.9284336Z ++ pwd 2024-06-26T07:10:29.9285138Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:29.9285820Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:29.9295982Z + local name=inference 2024-06-26T07:10:29.9296567Z + shift 2024-06-26T07:10:29.9296982Z + local suite=huggingface 2024-06-26T07:10:29.9297498Z + shift 2024-06-26T07:10:29.9297881Z + local shard_id=0 2024-06-26T07:10:29.9298227Z + shift 2024-06-26T07:10:29.9298508Z + partition_flags=() 2024-06-26T07:10:29.9298859Z + local partition_flags 2024-06-26T07:10:29.9299366Z + [[ -n 1 ]] 2024-06-26T07:10:29.9299870Z + [[ -n 0 ]] 2024-06-26T07:10:29.9300468Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:10:29.9301177Z + [[ aot_inductor_huggingface == *perf_compare* ]] 2024-06-26T07:10:29.9301677Z + [[ aot_inductor_huggingface == *perf* ]] 2024-06-26T07:10:29.9302180Z + [[ aot_inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:10:29.9302684Z + export TORCHINDUCTOR_ABI_COMPATIBLE=1 2024-06-26T07:10:29.9303147Z + TORCHINDUCTOR_ABI_COMPATIBLE=1 2024-06-26T07:10:29.9304779Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --export-aot-inductor --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-06-26T07:10:32.5861949Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:10:32.5863649Z warnings.warn( 2024-06-26T07:10:32.6296334Z 2024-06-26T07:10:32.6296951Z config.json: 0% 0.00/694 [00:00at(28); 2024-06-26T07:11:14.0076144Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:11:17.7586249Z pass 2024-06-26T07:11:17.7745009Z TIMING: backend_compile:0.00014 entire_frame_compile:1.21382 2024-06-26T07:11:17.7746377Z STATS: call_* op count: 439 | FakeTensorMode.__torch_dispatch__:22111 | FakeTensor.__torch_dispatch__:3105 | ProxyTorchDispatchMode.__torch_dispatch__:6214 2024-06-26T07:11:17.7747531Z Dynamo produced 1 graphs covering 439 ops with 0 graph breaks (0 unique) 2024-06-26T07:11:21.7940578Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:11:21.7942251Z warnings.warn( 2024-06-26T07:11:21.9914091Z 2024-06-26T07:11:24.5222076Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:11:24.5222562Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:11:24.5230785Z cuda eval AlbertForQuestionAnswering 2024-06-26T07:11:57.5193631Z /tmp/tmpbg5lqifs/cyghmathzjdinodveqim5elyll7r4ogl372c5qmpgrnb7hgacvbr/cqv7x5uj2mzugh26aipbqiycqkrviz3ju763mbvaka7i7w46n7te.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:11:57.5198371Z /tmp/tmpbg5lqifs/cyghmathzjdinodveqim5elyll7r4ogl372c5qmpgrnb7hgacvbr/cqv7x5uj2mzugh26aipbqiycqkrviz3ju763mbvaka7i7w46n7te.cpp:857:10: warning: variable ‘albert_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:11:57.5200115Z 857 | auto albert_embeddings_token_type_ids = constants_->at(24); 2024-06-26T07:11:57.5200693Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:12:01.2420056Z pass 2024-06-26T07:12:01.2556092Z TIMING: backend_compile:0.00014 entire_frame_compile:1.29469 2024-06-26T07:12:01.2557610Z STATS: call_* op count: 439 | FakeTensorMode.__torch_dispatch__:22077 | FakeTensor.__torch_dispatch__:3090 | ProxyTorchDispatchMode.__torch_dispatch__:6204 2024-06-26T07:12:01.2558893Z Dynamo produced 1 graphs covering 439 ops with 0 graph breaks (0 unique) 2024-06-26T07:12:05.1669583Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:12:05.1671286Z warnings.warn( 2024-06-26T07:12:05.4782932Z 2024-06-26T07:12:07.9672572Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:12:07.9673215Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:12:07.9744569Z cuda eval AllenaiLongformerBase 2024-06-26T07:12:11.4408363Z E0626 07:12:11.440000 140136492814976 torch/fx/experimental/recording.py:281] [0/0] failed while running evaluate_expr(*(Eq(u0, 1), None), **{'fx_node': u0}) 2024-06-26T07:12:11.4840827Z ERROR:common: 2024-06-26T07:12:11.4841174Z Traceback (most recent call last): 2024-06-26T07:12:11.4842349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1040, in evaluate_expr 2024-06-26T07:12:11.4843490Z return guard_scalar(self.sym_num) 2024-06-26T07:12:11.4844774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 788, in guard_scalar 2024-06-26T07:12:11.4845856Z return guard_bool(a) 2024-06-26T07:12:11.4847249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 961, in guard_bool 2024-06-26T07:12:11.4848610Z return a.node.guard_bool("", 0) # NB: uses Python backtrace 2024-06-26T07:12:11.4850007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 414, in guard_bool 2024-06-26T07:12:11.4851347Z r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) 2024-06-26T07:12:11.4853064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/recording.py", line 275, in wrapper 2024-06-26T07:12:11.4854269Z return event.run(self) 2024-06-26T07:12:11.4855422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/recording.py", line 162, in run 2024-06-26T07:12:11.4856268Z return self.f(*args, **kwargs) 2024-06-26T07:12:11.4857275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5147, in evaluate_expr 2024-06-26T07:12:11.4858227Z concrete_val = compute_concrete_val() 2024-06-26T07:12:11.4859309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5128, in compute_concrete_val 2024-06-26T07:12:11.4860277Z return self.size_hint(orig_expr) 2024-06-26T07:12:11.4861281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4666, in size_hint 2024-06-26T07:12:11.4862277Z raise self._make_data_dependent_error(result_expr, expr) 2024-06-26T07:12:11.4863607Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Eq(u0, 1) (unhinted: Eq(u0, 1)). (Size-like symbols: none) 2024-06-26T07:12:11.4864587Z 2024-06-26T07:12:11.4864865Z Potential framework code culprit (scroll up for full backtrace): 2024-06-26T07:12:11.4865970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1040, in evaluate_expr 2024-06-26T07:12:11.4866864Z return guard_scalar(self.sym_num) 2024-06-26T07:12:11.4867155Z 2024-06-26T07:12:11.4867361Z For more information, run with TORCH_LOGS="dynamic" 2024-06-26T07:12:11.4868154Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-06-26T07:12:11.4869176Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-06-26T07:12:11.4870373Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-06-26T07:12:11.4871161Z 2024-06-26T07:12:11.4871319Z User Stack (most recent call last): 2024-06-26T07:12:11.4871757Z (snipped, see stack below for prefix) 2024-06-26T07:12:11.4872838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-06-26T07:12:11.4874186Z outputs = self.longformer( 2024-06-26T07:12:11.4875094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.4875929Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.4877001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-06-26T07:12:11.4877967Z encoder_outputs = self.encoder( 2024-06-26T07:12:11.4878879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.4879723Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.4880796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-06-26T07:12:11.4881754Z layer_outputs = layer_module( 2024-06-26T07:12:11.4882646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.4883606Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.4884782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-06-26T07:12:11.4885753Z self_attn_outputs = self.attention( 2024-06-26T07:12:11.4886675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.4887597Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.4888662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-06-26T07:12:11.4889616Z self_outputs = self.self( 2024-06-26T07:12:11.4890489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.4891327Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.4892397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-06-26T07:12:11.4893347Z if is_global_attn: 2024-06-26T07:12:11.4893563Z 2024-06-26T07:12:11.4893826Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-06-26T07:12:11.4894250Z 2024-06-26T07:12:11.4894545Z During handling of the above exception, another exception occurred: 2024-06-26T07:12:11.4895006Z 2024-06-26T07:12:11.4895155Z Traceback (most recent call last): 2024-06-26T07:12:11.4895884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2616, in check_accuracy 2024-06-26T07:12:11.4896641Z optimized_model_iter_fn = optimize_ctx( 2024-06-26T07:12:11.4897438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1222, in export_aot_inductor 2024-06-26T07:12:11.4898378Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-06-26T07:12:11.4899236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1192, in load 2024-06-26T07:12:11.4899971Z gm = torch.export._trace._export_to_torch_ir( 2024-06-26T07:12:11.4900958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-06-26T07:12:11.4901849Z gm_torch_level, _ = torch._dynamo.export( 2024-06-26T07:12:11.4902771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1384, in inner 2024-06-26T07:12:11.4903593Z result_traced = opt_f(*args, **kwargs) 2024-06-26T07:12:11.4904584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:12:11.4905495Z return self._call_impl(*args, **kwargs) 2024-06-26T07:12:11.4906431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.4907277Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.4908151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 434, in _fn 2024-06-26T07:12:11.4908937Z return fn(*args, **kwargs) 2024-06-26T07:12:11.4909869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:12:11.4910777Z return self._call_impl(*args, **kwargs) 2024-06-26T07:12:11.4911712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.4912556Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.4913487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1121, in __call__ 2024-06-26T07:12:11.4914573Z return self._torchdynamo_orig_callable( 2024-06-26T07:12:11.4915506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 472, in __call__ 2024-06-26T07:12:11.4916309Z return _compile( 2024-06-26T07:12:11.4917248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 85, in wrapper_function 2024-06-26T07:12:11.4918228Z return StrobelightCompileTimeProfiler.profile_compile_time( 2024-06-26T07:12:11.4919418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time 2024-06-26T07:12:11.4920372Z return func(*args, **kwargs) 2024-06-26T07:12:11.4921051Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T07:12:11.4921691Z return func(*args, **kwds) 2024-06-26T07:12:11.4922552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 817, in _compile 2024-06-26T07:12:11.4923506Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-06-26T07:12:11.4924591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T07:12:11.4925396Z r = func(*args, **kwargs) 2024-06-26T07:12:11.4926290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 636, in compile_inner 2024-06-26T07:12:11.4927196Z out_code = transform_code_object(code, transform) 2024-06-26T07:12:11.4928305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1270, in transform_code_object 2024-06-26T07:12:11.4929363Z transformations(instructions, code_options) 2024-06-26T07:12:11.4930288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 178, in _fn 2024-06-26T07:12:11.4931077Z return fn(*args, **kwargs) 2024-06-26T07:12:11.4931951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 582, in transform 2024-06-26T07:12:11.4932756Z tracer.run() 2024-06-26T07:12:11.4933553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2476, in run 2024-06-26T07:12:11.4934341Z super().run() 2024-06-26T07:12:11.4935140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.4935941Z while self.step(): 2024-06-26T07:12:11.4936764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.4937625Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.4938621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.4939480Z return inner_fn(self, inst) 2024-06-26T07:12:11.4940430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1533, in CALL_FUNCTION_KW 2024-06-26T07:12:11.4941328Z self.call_function(fn, args, kwargs) 2024-06-26T07:12:11.4942291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.4943211Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.4944239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 433, in call_function 2024-06-26T07:12:11.4945139Z return tx.inline_user_function_return( 2024-06-26T07:12:11.4946190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.4947318Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.4948459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.4949392Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.4950412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.4951341Z tracer.run() 2024-06-26T07:12:11.4952140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.4952939Z while self.step(): 2024-06-26T07:12:11.4953916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.4954828Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.4955929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.4956763Z return inner_fn(self, inst) 2024-06-26T07:12:11.4957708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1521, in CALL_FUNCTION_EX 2024-06-26T07:12:11.4958678Z self.call_function(fn, argsvars.items, kwargsvars) 2024-06-26T07:12:11.4959712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.4960646Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.4961690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 353, in call_function 2024-06-26T07:12:11.4962617Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.4963652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 295, in call_function 2024-06-26T07:12:11.4964666Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.4965682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 95, in call_function 2024-06-26T07:12:11.4966751Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-06-26T07:12:11.4967975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.4969104Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.4970252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.4971183Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.4972195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.4973039Z tracer.run() 2024-06-26T07:12:11.4973830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.4974627Z while self.step(): 2024-06-26T07:12:11.4975443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.4976312Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.4977271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.4978099Z return inner_fn(self, inst) 2024-06-26T07:12:11.4979085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1533, in CALL_FUNCTION_KW 2024-06-26T07:12:11.4979983Z self.call_function(fn, args, kwargs) 2024-06-26T07:12:11.4980953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.4981864Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.4982883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 433, in call_function 2024-06-26T07:12:11.4983782Z return tx.inline_user_function_return( 2024-06-26T07:12:11.4984824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.4986032Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.4987177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.4988102Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.4989122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.4990002Z tracer.run() 2024-06-26T07:12:11.4990794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.4991581Z while self.step(): 2024-06-26T07:12:11.4992406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.4993276Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.4994448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.4995288Z return inner_fn(self, inst) 2024-06-26T07:12:11.4996248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1521, in CALL_FUNCTION_EX 2024-06-26T07:12:11.4997208Z self.call_function(fn, argsvars.items, kwargsvars) 2024-06-26T07:12:11.4998245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.4999155Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.5000194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 353, in call_function 2024-06-26T07:12:11.5001133Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5002164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 295, in call_function 2024-06-26T07:12:11.5003101Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5004113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 95, in call_function 2024-06-26T07:12:11.5005277Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-06-26T07:12:11.5006515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.5007645Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.5008789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.5009733Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.5010766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.5011616Z tracer.run() 2024-06-26T07:12:11.5012406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.5013214Z while self.step(): 2024-06-26T07:12:11.5014045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.5014919Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.5015896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.5016722Z return inner_fn(self, inst) 2024-06-26T07:12:11.5017674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1533, in CALL_FUNCTION_KW 2024-06-26T07:12:11.5018572Z self.call_function(fn, args, kwargs) 2024-06-26T07:12:11.5019538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.5020591Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.5021624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 433, in call_function 2024-06-26T07:12:11.5022530Z return tx.inline_user_function_return( 2024-06-26T07:12:11.5023581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.5024766Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.5025927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.5026867Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.5027895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.5028740Z tracer.run() 2024-06-26T07:12:11.5029540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.5030342Z while self.step(): 2024-06-26T07:12:11.5031172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.5032043Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.5033009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.5034081Z return inner_fn(self, inst) 2024-06-26T07:12:11.5035046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1521, in CALL_FUNCTION_EX 2024-06-26T07:12:11.5036005Z self.call_function(fn, argsvars.items, kwargsvars) 2024-06-26T07:12:11.5037026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.5037952Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.5038987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 353, in call_function 2024-06-26T07:12:11.5039920Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5040935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 295, in call_function 2024-06-26T07:12:11.5041873Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5042891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 95, in call_function 2024-06-26T07:12:11.5043961Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-06-26T07:12:11.5045297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.5046423Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.5047573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.5048516Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.5049540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.5050374Z tracer.run() 2024-06-26T07:12:11.5051173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.5051967Z while self.step(): 2024-06-26T07:12:11.5052792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.5053645Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.5054610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.5055544Z return inner_fn(self, inst) 2024-06-26T07:12:11.5056492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1533, in CALL_FUNCTION_KW 2024-06-26T07:12:11.5057390Z self.call_function(fn, args, kwargs) 2024-06-26T07:12:11.5058348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.5059336Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.5060374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 433, in call_function 2024-06-26T07:12:11.5061278Z return tx.inline_user_function_return( 2024-06-26T07:12:11.5062326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.5063449Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.5064599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.5065530Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.5066545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.5067392Z tracer.run() 2024-06-26T07:12:11.5068194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.5068995Z while self.step(): 2024-06-26T07:12:11.5069811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.5070675Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.5071640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.5072476Z return inner_fn(self, inst) 2024-06-26T07:12:11.5073426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1521, in CALL_FUNCTION_EX 2024-06-26T07:12:11.5074593Z self.call_function(fn, argsvars.items, kwargsvars) 2024-06-26T07:12:11.5075723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.5076743Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.5077899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 353, in call_function 2024-06-26T07:12:11.5078921Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5080064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 295, in call_function 2024-06-26T07:12:11.5081081Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5082225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 95, in call_function 2024-06-26T07:12:11.5083412Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-06-26T07:12:11.5084874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.5086017Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.5087174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.5088115Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.5089197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.5090047Z tracer.run() 2024-06-26T07:12:11.5090850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.5091751Z while self.step(): 2024-06-26T07:12:11.5092578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.5093454Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.5094424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.5095316Z return inner_fn(self, inst) 2024-06-26T07:12:11.5096268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1533, in CALL_FUNCTION_KW 2024-06-26T07:12:11.5097169Z self.call_function(fn, args, kwargs) 2024-06-26T07:12:11.5098135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.5099104Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.5100140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 433, in call_function 2024-06-26T07:12:11.5101044Z return tx.inline_user_function_return( 2024-06-26T07:12:11.5102101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.5103232Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.5104389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.5105318Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.5106357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.5107201Z tracer.run() 2024-06-26T07:12:11.5108003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.5108802Z while self.step(): 2024-06-26T07:12:11.5109634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.5110506Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.5111474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 510, in wrapper 2024-06-26T07:12:11.5112302Z return inner_fn(self, inst) 2024-06-26T07:12:11.5113259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1521, in CALL_FUNCTION_EX 2024-06-26T07:12:11.5114440Z self.call_function(fn, argsvars.items, kwargsvars) 2024-06-26T07:12:11.5115482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 754, in call_function 2024-06-26T07:12:11.5116404Z self.push(fn.call_function(self, args, kwargs)) 2024-06-26T07:12:11.5117451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 353, in call_function 2024-06-26T07:12:11.5118388Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5119427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 295, in call_function 2024-06-26T07:12:11.5120361Z return super().call_function(tx, args, kwargs) 2024-06-26T07:12:11.5121383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 95, in call_function 2024-06-26T07:12:11.5122458Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-06-26T07:12:11.5123698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 760, in inline_user_function_return 2024-06-26T07:12:11.5124953Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-06-26T07:12:11.5126196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2691, in inline_call 2024-06-26T07:12:11.5127130Z return cls.inline_call_(parent, func, args, kwargs) 2024-06-26T07:12:11.5128155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2807, in inline_call_ 2024-06-26T07:12:11.5129041Z tracer.run() 2024-06-26T07:12:11.5129888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T07:12:11.5130679Z while self.step(): 2024-06-26T07:12:11.5131510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T07:12:11.5132381Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T07:12:11.5133333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 464, in inner 2024-06-26T07:12:11.5134205Z eval_result = value.evaluate_expr(self.output) 2024-06-26T07:12:11.5135232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1042, in evaluate_expr 2024-06-26T07:12:11.5136111Z raise UserError( # noqa: B904 2024-06-26T07:12:11.5137366Z torch._dynamo.exc.UserError: Consider annotating your code using torch._check*(). Could not guard on data-dependent expression Eq(u0, 1) (unhinted: Eq(u0, 1)). (Size-like symbols: none) 2024-06-26T07:12:11.5138363Z 2024-06-26T07:12:11.5138656Z Potential framework code culprit (scroll up for full backtrace): 2024-06-26T07:12:11.5139768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1040, in evaluate_expr 2024-06-26T07:12:11.5140657Z return guard_scalar(self.sym_num) 2024-06-26T07:12:11.5140949Z 2024-06-26T07:12:11.5141161Z For more information, run with TORCH_LOGS="dynamic" 2024-06-26T07:12:11.5141962Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-06-26T07:12:11.5142922Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-06-26T07:12:11.5144116Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-06-26T07:12:11.5144921Z 2024-06-26T07:12:11.5145073Z User Stack (most recent call last): 2024-06-26T07:12:11.5145521Z (snipped, see stack below for prefix) 2024-06-26T07:12:11.5146604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-06-26T07:12:11.5147578Z outputs = self.longformer( 2024-06-26T07:12:11.5148471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5149316Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5150388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-06-26T07:12:11.5151364Z encoder_outputs = self.encoder( 2024-06-26T07:12:11.5152273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5153115Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5154419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-06-26T07:12:11.5155393Z layer_outputs = layer_module( 2024-06-26T07:12:11.5156304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5157150Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5158235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-06-26T07:12:11.5159260Z self_attn_outputs = self.attention( 2024-06-26T07:12:11.5160289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5161142Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5162221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-06-26T07:12:11.5163181Z self_outputs = self.self( 2024-06-26T07:12:11.5164129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5165066Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5166132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-06-26T07:12:11.5167082Z if is_global_attn: 2024-06-26T07:12:11.5167301Z 2024-06-26T07:12:11.5167560Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-06-26T07:12:11.5168713Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-06-26T07:12:11.5169501Z 2024-06-26T07:12:11.5169624Z from user code: 2024-06-26T07:12:11.5170600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-06-26T07:12:11.5171561Z outputs = self.longformer( 2024-06-26T07:12:11.5172454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5173300Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5174367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-06-26T07:12:11.5175332Z encoder_outputs = self.encoder( 2024-06-26T07:12:11.5176237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5177084Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5188611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in forward 2024-06-26T07:12:11.5189617Z layer_outputs = layer_module( 2024-06-26T07:12:11.5190506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5191351Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5192423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-06-26T07:12:11.5193400Z self_attn_outputs = self.attention( 2024-06-26T07:12:11.5194444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5195288Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5196351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-06-26T07:12:11.5197311Z self_outputs = self.self( 2024-06-26T07:12:11.5198166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:12:11.5199004Z return forward_call(*args, **kwargs) 2024-06-26T07:12:11.5200065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 601, in forward 2024-06-26T07:12:11.5201002Z if is_global_attn: 2024-06-26T07:12:11.5201216Z 2024-06-26T07:12:11.5201524Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-06-26T07:12:11.5201976Z 2024-06-26T07:12:11.5202274Z TorchDynamo optimized model failed to run because of following error 2024-06-26T07:12:11.5202847Z fail_to_run 2024-06-26T07:12:11.5203125Z TIMING: 2024-06-26T07:12:11.5203709Z STATS: call_* op count: 0 | FakeTensorMode.__torch_dispatch__:514 | FakeTensor.__torch_dispatch__:38 2024-06-26T07:12:11.5204829Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:12:14.8696426Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:12:14.8698115Z warnings.warn( 2024-06-26T07:12:15.0630821Z 2024-06-26T07:12:18.6156551Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:12:18.6157036Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:12:18.6235057Z cuda eval BartForCausalLM 2024-06-26T07:12:47.5181309Z pass 2024-06-26T07:12:47.5465419Z TIMING: backend_compile:0.00012 entire_frame_compile:1.09776 2024-06-26T07:12:47.5467896Z STATS: call_* op count: 328 | FakeTensorMode.__torch_dispatch__:21628 | FakeTensor.__torch_dispatch__:2980 | ProxyTorchDispatchMode.__torch_dispatch__:5843 2024-06-26T07:12:47.5469099Z Dynamo produced 1 graphs covering 328 ops with 0 graph breaks (0 unique) 2024-06-26T07:12:51.4815137Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:12:51.4816810Z warnings.warn( 2024-06-26T07:12:51.7113312Z 2024-06-26T07:12:58.1316900Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:12:58.1317757Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:12:58.1449360Z cuda eval BartForConditionalGeneration 2024-06-26T07:14:00.4896956Z pass 2024-06-26T07:14:00.5262344Z TIMING: backend_compile:0.00023 entire_frame_compile:2.71522 2024-06-26T07:14:00.5264206Z STATS: call_* op count: 888 | FakeTensorMode.__torch_dispatch__:56160 | FakeTensor.__torch_dispatch__:7698 | ProxyTorchDispatchMode.__torch_dispatch__:15364 2024-06-26T07:14:00.5265388Z Dynamo produced 1 graphs covering 888 ops with 0 graph breaks (0 unique) 2024-06-26T07:14:05.1159617Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:14:05.1161341Z warnings.warn( 2024-06-26T07:14:05.3420938Z 2024-06-26T07:14:07.0992512Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:14:07.0992989Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:14:07.1049098Z cuda eval BertForMaskedLM 2024-06-26T07:14:32.4976154Z /tmp/tmpbg5lqifs/c5uddbj35v7hj6qndo6x25j76ppeml5n6o4kphxzkqm5xo4ajyaa/cvhokondm4kiknntjlqrierc6p5vvwgb3rkhw63itrpdusl24neb.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:14:32.4979541Z /tmp/tmpbg5lqifs/c5uddbj35v7hj6qndo6x25j76ppeml5n6o4kphxzkqm5xo4ajyaa/cvhokondm4kiknntjlqrierc6p5vvwgb3rkhw63itrpdusl24neb.cpp:2633:10: warning: variable ‘bert_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:14:32.4981350Z 2633 | auto bert_embeddings_token_type_ids = constants_->at(202); 2024-06-26T07:14:32.4981986Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:14:34.7164386Z pass 2024-06-26T07:14:34.7312206Z TIMING: backend_compile:0.00014 entire_frame_compile:2.65568 2024-06-26T07:14:34.7313491Z STATS: call_* op count: 370 | FakeTensorMode.__torch_dispatch__:23977 | FakeTensor.__torch_dispatch__:3086 | ProxyTorchDispatchMode.__torch_dispatch__:6788 2024-06-26T07:14:34.7314838Z Dynamo produced 1 graphs covering 370 ops with 0 graph breaks (0 unique) 2024-06-26T07:14:38.7377483Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:14:38.7379438Z warnings.warn( 2024-06-26T07:14:38.9524604Z 2024-06-26T07:14:40.4731485Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:14:40.4731938Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:14:40.4787514Z cuda eval BertForQuestionAnswering 2024-06-26T07:15:05.5729350Z /tmp/tmpbg5lqifs/chxaei7wfgtlbci7enz65tuyrqo7erythwhvod3g4wzawdgss5ew/csbc3h6kmrlvvzlltyiqjfzvfrdwzpatqxewgabhgpphnpv45lfp.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:15:05.5732550Z /tmp/tmpbg5lqifs/chxaei7wfgtlbci7enz65tuyrqo7erythwhvod3g4wzawdgss5ew/csbc3h6kmrlvvzlltyiqjfzvfrdwzpatqxewgabhgpphnpv45lfp.cpp:2596:10: warning: variable ‘bert_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:15:05.5734279Z 2596 | auto bert_embeddings_token_type_ids = constants_->at(198); 2024-06-26T07:15:05.5734847Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:15:07.7507838Z pass 2024-06-26T07:15:07.7633673Z TIMING: backend_compile:0.00014 entire_frame_compile:2.65289 2024-06-26T07:15:07.7635036Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:24057 | FakeTensor.__torch_dispatch__:3071 | ProxyTorchDispatchMode.__torch_dispatch__:6802 2024-06-26T07:15:07.7636210Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2024-06-26T07:15:11.7127394Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:15:11.7129081Z warnings.warn( 2024-06-26T07:15:11.9240515Z 2024-06-26T07:15:40.6892929Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:15:40.6895152Z loading model: 0it [00:28, ?it/s] 2024-06-26T07:15:40.7049745Z cuda eval BlenderbotForCausalLM 2024-06-26T07:15:40.7052652Z pass_due_to_skip 2024-06-26T07:15:41.1319964Z TIMING: 2024-06-26T07:15:41.1320330Z STATS: call_* op count: 0 2024-06-26T07:15:41.1324513Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:15:44.3773907Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:15:44.3775617Z warnings.warn( 2024-06-26T07:15:44.6192868Z 2024-06-26T07:15:45.8224509Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:15:45.8225401Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:15:45.8275093Z cuda eval BlenderbotSmallForCausalLM 2024-06-26T07:15:45.8477340Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-06-26T07:16:06.7816950Z pass 2024-06-26T07:16:06.7947847Z TIMING: backend_compile:0.00012 entire_frame_compile:0.87224 2024-06-26T07:16:06.7948956Z STATS: call_* op count: 326 | FakeTensorMode.__torch_dispatch__:16737 | FakeTensor.__torch_dispatch__:2154 | ProxyTorchDispatchMode.__torch_dispatch__:4501 2024-06-26T07:16:06.7950073Z Dynamo produced 1 graphs covering 326 ops with 0 graph breaks (0 unique) 2024-06-26T07:16:10.6169374Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:16:10.6171090Z warnings.warn( 2024-06-26T07:16:10.8281586Z 2024-06-26T07:16:12.3898053Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:16:12.3898681Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:16:12.4002177Z cuda eval BlenderbotSmallForConditionalGeneration 2024-06-26T07:16:12.5406985Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:16:52.6853996Z pass 2024-06-26T07:16:52.6931608Z TIMING: backend_compile:0.00022 entire_frame_compile:2.11493 2024-06-26T07:16:52.6933969Z STATS: call_* op count: 838 | FakeTensorMode.__torch_dispatch__:42591 | FakeTensor.__torch_dispatch__:5503 | ProxyTorchDispatchMode.__torch_dispatch__:11850 2024-06-26T07:16:52.6935351Z Dynamo produced 1 graphs covering 838 ops with 0 graph breaks (0 unique) 2024-06-26T07:16:56.9550475Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:16:56.9552240Z warnings.warn( 2024-06-26T07:16:57.1997241Z 2024-06-26T07:16:58.9928774Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:16:58.9929328Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:16:58.9980458Z cuda eval CamemBert 2024-06-26T07:17:24.3833601Z /tmp/tmpbg5lqifs/cqafldaaokgsw3gjnjcixjui7lusgsbqimlhg6svbvzjuamxjmti/cord7hyy6ict55xh4mf4ggu2gxluoc7auxnc4wibe6ulnqmuu3ls.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:17:24.3840192Z /tmp/tmpbg5lqifs/cqafldaaokgsw3gjnjcixjui7lusgsbqimlhg6svbvzjuamxjmti/cord7hyy6ict55xh4mf4ggu2gxluoc7auxnc4wibe6ulnqmuu3ls.cpp:2625:10: warning: variable ‘roberta_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:24.3842570Z 2625 | auto roberta_embeddings_token_type_ids = constants_->at(202); 2024-06-26T07:17:24.3843343Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:26.6179530Z pass 2024-06-26T07:17:26.6353192Z TIMING: backend_compile:0.00013 entire_frame_compile:2.64692 2024-06-26T07:17:26.6354805Z STATS: call_* op count: 378 | FakeTensorMode.__torch_dispatch__:24110 | FakeTensor.__torch_dispatch__:3111 | ProxyTorchDispatchMode.__torch_dispatch__:6833 2024-06-26T07:17:26.6355969Z Dynamo produced 1 graphs covering 378 ops with 0 graph breaks (0 unique) 2024-06-26T07:17:30.6396927Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:17:30.6398877Z warnings.warn( 2024-06-26T07:17:30.8481121Z 2024-06-26T07:17:32.9711495Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:17:32.9711977Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:17:32.9755738Z cuda eval DebertaForMaskedLM 2024-06-26T07:17:58.4124659Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:17:58.4128387Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2092:10: warning: variable ‘deberta_encoder_layer_0_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4136175Z 2092 | auto deberta_encoder_layer_0_attention_self_q_bias = constants_->at(2); 2024-06-26T07:17:58.4136970Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4139285Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2093:10: warning: variable ‘deberta_encoder_layer_0_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4141336Z 2093 | auto deberta_encoder_layer_0_attention_self_v_bias = constants_->at(3); 2024-06-26T07:17:58.4141995Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4143743Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2098:10: warning: variable ‘deberta_encoder_layer_1_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4145681Z 2098 | auto deberta_encoder_layer_1_attention_self_q_bias = constants_->at(8); 2024-06-26T07:17:58.4146329Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4148060Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2099:10: warning: variable ‘deberta_encoder_layer_1_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4149843Z 2099 | auto deberta_encoder_layer_1_attention_self_v_bias = constants_->at(9); 2024-06-26T07:17:58.4150520Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4152239Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2104:10: warning: variable ‘deberta_encoder_layer_2_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4154355Z 2104 | auto deberta_encoder_layer_2_attention_self_q_bias = constants_->at(14); 2024-06-26T07:17:58.4155027Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4156769Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2105:10: warning: variable ‘deberta_encoder_layer_2_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4158556Z 2105 | auto deberta_encoder_layer_2_attention_self_v_bias = constants_->at(15); 2024-06-26T07:17:58.4159233Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4160966Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2110:10: warning: variable ‘deberta_encoder_layer_3_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4162751Z 2110 | auto deberta_encoder_layer_3_attention_self_q_bias = constants_->at(20); 2024-06-26T07:17:58.4163412Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4165273Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2111:10: warning: variable ‘deberta_encoder_layer_3_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4167050Z 2111 | auto deberta_encoder_layer_3_attention_self_v_bias = constants_->at(21); 2024-06-26T07:17:58.4167696Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4169422Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2116:10: warning: variable ‘deberta_encoder_layer_4_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4171206Z 2116 | auto deberta_encoder_layer_4_attention_self_q_bias = constants_->at(26); 2024-06-26T07:17:58.4171858Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4173587Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2117:10: warning: variable ‘deberta_encoder_layer_4_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4175392Z 2117 | auto deberta_encoder_layer_4_attention_self_v_bias = constants_->at(27); 2024-06-26T07:17:58.4176060Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4177776Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2122:10: warning: variable ‘deberta_encoder_layer_5_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4179681Z 2122 | auto deberta_encoder_layer_5_attention_self_q_bias = constants_->at(32); 2024-06-26T07:17:58.4180331Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4182125Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2123:10: warning: variable ‘deberta_encoder_layer_5_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4183910Z 2123 | auto deberta_encoder_layer_5_attention_self_v_bias = constants_->at(33); 2024-06-26T07:17:58.4184556Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4186272Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2128:10: warning: variable ‘deberta_encoder_layer_6_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4188047Z 2128 | auto deberta_encoder_layer_6_attention_self_q_bias = constants_->at(38); 2024-06-26T07:17:58.4188699Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4190428Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2129:10: warning: variable ‘deberta_encoder_layer_6_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4192201Z 2129 | auto deberta_encoder_layer_6_attention_self_v_bias = constants_->at(39); 2024-06-26T07:17:58.4192848Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4194794Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2134:10: warning: variable ‘deberta_encoder_layer_7_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4196589Z 2134 | auto deberta_encoder_layer_7_attention_self_q_bias = constants_->at(44); 2024-06-26T07:17:58.4197243Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4198981Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2135:10: warning: variable ‘deberta_encoder_layer_7_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4200757Z 2135 | auto deberta_encoder_layer_7_attention_self_v_bias = constants_->at(45); 2024-06-26T07:17:58.4201411Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4203126Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2140:10: warning: variable ‘deberta_encoder_layer_8_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4204982Z 2140 | auto deberta_encoder_layer_8_attention_self_q_bias = constants_->at(50); 2024-06-26T07:17:58.4205657Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4207400Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2141:10: warning: variable ‘deberta_encoder_layer_8_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4209190Z 2141 | auto deberta_encoder_layer_8_attention_self_v_bias = constants_->at(51); 2024-06-26T07:17:58.4209847Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4211568Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2146:10: warning: variable ‘deberta_encoder_layer_9_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4213458Z 2146 | auto deberta_encoder_layer_9_attention_self_q_bias = constants_->at(56); 2024-06-26T07:17:58.4214112Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4215834Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2147:10: warning: variable ‘deberta_encoder_layer_9_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4217697Z 2147 | auto deberta_encoder_layer_9_attention_self_v_bias = constants_->at(57); 2024-06-26T07:17:58.4218368Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4220097Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2152:10: warning: variable ‘deberta_encoder_layer_10_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4221882Z 2152 | auto deberta_encoder_layer_10_attention_self_q_bias = constants_->at(62); 2024-06-26T07:17:58.4222562Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4224295Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2153:10: warning: variable ‘deberta_encoder_layer_10_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4226081Z 2153 | auto deberta_encoder_layer_10_attention_self_v_bias = constants_->at(63); 2024-06-26T07:17:58.4226744Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4228484Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2158:10: warning: variable ‘deberta_encoder_layer_11_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4230276Z 2158 | auto deberta_encoder_layer_11_attention_self_q_bias = constants_->at(68); 2024-06-26T07:17:58.4230948Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:17:58.4232679Z /tmp/tmpbg5lqifs/czxphjwqq52lp66j5qrhyllyrz3pwn4jmkr5xy6wovu4cuevxcqq/cwerbr3dioqwidthygfjn7wvqr34vcdic2hgqb4wcid376lm4ywr.cpp:2159:10: warning: variable ‘deberta_encoder_layer_11_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:17:58.4234659Z 2159 | auto deberta_encoder_layer_11_attention_self_v_bias = constants_->at(69); 2024-06-26T07:17:58.4235336Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:00.8441533Z pass 2024-06-26T07:18:00.8674316Z TIMING: backend_compile:0.00022 entire_frame_compile:2.38934 2024-06-26T07:18:00.8675557Z STATS: call_* op count: 790 | FakeTensorMode.__torch_dispatch__:26449 | FakeTensor.__torch_dispatch__:3018 | ProxyTorchDispatchMode.__torch_dispatch__:7250 2024-06-26T07:18:00.8676768Z Dynamo produced 1 graphs covering 790 ops with 0 graph breaks (0 unique) 2024-06-26T07:18:04.8072376Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:18:04.8074319Z warnings.warn( 2024-06-26T07:18:05.0068165Z 2024-06-26T07:18:06.7268467Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:18:06.7269099Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:18:06.7311205Z cuda eval DebertaForQuestionAnswering 2024-06-26T07:18:31.0662510Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:18:31.0666331Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2059:10: warning: variable ‘deberta_encoder_layer_0_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0668495Z 2059 | auto deberta_encoder_layer_0_attention_self_q_bias = constants_->at(2); 2024-06-26T07:18:31.0669270Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0671759Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2060:10: warning: variable ‘deberta_encoder_layer_0_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0674180Z 2060 | auto deberta_encoder_layer_0_attention_self_v_bias = constants_->at(3); 2024-06-26T07:18:31.0674837Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0676571Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2065:10: warning: variable ‘deberta_encoder_layer_1_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0678357Z 2065 | auto deberta_encoder_layer_1_attention_self_q_bias = constants_->at(8); 2024-06-26T07:18:31.0679000Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0680718Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2066:10: warning: variable ‘deberta_encoder_layer_1_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0682481Z 2066 | auto deberta_encoder_layer_1_attention_self_v_bias = constants_->at(9); 2024-06-26T07:18:31.0683143Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0685038Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2071:10: warning: variable ‘deberta_encoder_layer_2_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0686831Z 2071 | auto deberta_encoder_layer_2_attention_self_q_bias = constants_->at(14); 2024-06-26T07:18:31.0687484Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0689186Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2072:10: warning: variable ‘deberta_encoder_layer_2_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0690953Z 2072 | auto deberta_encoder_layer_2_attention_self_v_bias = constants_->at(15); 2024-06-26T07:18:31.0691610Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0693322Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2077:10: warning: variable ‘deberta_encoder_layer_3_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0695075Z 2077 | auto deberta_encoder_layer_3_attention_self_q_bias = constants_->at(20); 2024-06-26T07:18:31.0695731Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0697443Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2078:10: warning: variable ‘deberta_encoder_layer_3_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0699208Z 2078 | auto deberta_encoder_layer_3_attention_self_v_bias = constants_->at(21); 2024-06-26T07:18:31.0699859Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0701559Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2083:10: warning: variable ‘deberta_encoder_layer_4_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0703305Z 2083 | auto deberta_encoder_layer_4_attention_self_q_bias = constants_->at(26); 2024-06-26T07:18:31.0704080Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0705787Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2084:10: warning: variable ‘deberta_encoder_layer_4_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0707538Z 2084 | auto deberta_encoder_layer_4_attention_self_v_bias = constants_->at(27); 2024-06-26T07:18:31.0708264Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0709980Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2089:10: warning: variable ‘deberta_encoder_layer_5_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0711732Z 2089 | auto deberta_encoder_layer_5_attention_self_q_bias = constants_->at(32); 2024-06-26T07:18:31.0712385Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0714324Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2090:10: warning: variable ‘deberta_encoder_layer_5_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0716114Z 2090 | auto deberta_encoder_layer_5_attention_self_v_bias = constants_->at(33); 2024-06-26T07:18:31.0716776Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0718501Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2095:10: warning: variable ‘deberta_encoder_layer_6_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0720281Z 2095 | auto deberta_encoder_layer_6_attention_self_q_bias = constants_->at(38); 2024-06-26T07:18:31.0720938Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0722665Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2096:10: warning: variable ‘deberta_encoder_layer_6_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0724571Z 2096 | auto deberta_encoder_layer_6_attention_self_v_bias = constants_->at(39); 2024-06-26T07:18:31.0725223Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0726946Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2101:10: warning: variable ‘deberta_encoder_layer_7_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0728704Z 2101 | auto deberta_encoder_layer_7_attention_self_q_bias = constants_->at(44); 2024-06-26T07:18:31.0729356Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0731065Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2102:10: warning: variable ‘deberta_encoder_layer_7_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0732836Z 2102 | auto deberta_encoder_layer_7_attention_self_v_bias = constants_->at(45); 2024-06-26T07:18:31.0733491Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0735262Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2107:10: warning: variable ‘deberta_encoder_layer_8_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0737025Z 2107 | auto deberta_encoder_layer_8_attention_self_q_bias = constants_->at(50); 2024-06-26T07:18:31.0737681Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0739379Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2108:10: warning: variable ‘deberta_encoder_layer_8_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0741263Z 2108 | auto deberta_encoder_layer_8_attention_self_v_bias = constants_->at(51); 2024-06-26T07:18:31.0741914Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0743697Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2113:10: warning: variable ‘deberta_encoder_layer_9_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0745470Z 2113 | auto deberta_encoder_layer_9_attention_self_q_bias = constants_->at(56); 2024-06-26T07:18:31.0746127Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0747838Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2114:10: warning: variable ‘deberta_encoder_layer_9_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0749610Z 2114 | auto deberta_encoder_layer_9_attention_self_v_bias = constants_->at(57); 2024-06-26T07:18:31.0750268Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0752002Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2119:10: warning: variable ‘deberta_encoder_layer_10_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0753980Z 2119 | auto deberta_encoder_layer_10_attention_self_q_bias = constants_->at(62); 2024-06-26T07:18:31.0754684Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0756415Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2120:10: warning: variable ‘deberta_encoder_layer_10_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0758210Z 2120 | auto deberta_encoder_layer_10_attention_self_v_bias = constants_->at(63); 2024-06-26T07:18:31.0758877Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0760604Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2125:10: warning: variable ‘deberta_encoder_layer_11_attention_self_q_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0762402Z 2125 | auto deberta_encoder_layer_11_attention_self_q_bias = constants_->at(68); 2024-06-26T07:18:31.0763058Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:31.0764922Z /tmp/tmpbg5lqifs/c2vvcg5jd7wysm5h4bnw7ehp45732xvge4jvkhckmbohyiutf7c3/c6svwb2rmqlxrkk2ufcr5warw7ymtznmb6re6kwfoqh2gr2wwb6m.cpp:2126:10: warning: variable ‘deberta_encoder_layer_11_attention_self_v_bias’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:18:31.0766697Z 2126 | auto deberta_encoder_layer_11_attention_self_v_bias = constants_->at(69); 2024-06-26T07:18:31.0767369Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:18:33.4795835Z pass 2024-06-26T07:18:33.4879323Z TIMING: backend_compile:0.00023 entire_frame_compile:2.36092 2024-06-26T07:18:33.4880427Z STATS: call_* op count: 797 | FakeTensorMode.__torch_dispatch__:26529 | FakeTensor.__torch_dispatch__:3004 | ProxyTorchDispatchMode.__torch_dispatch__:7264 2024-06-26T07:18:33.4881596Z Dynamo produced 1 graphs covering 797 ops with 0 graph breaks (0 unique) 2024-06-26T07:18:37.4442740Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:18:37.4444490Z warnings.warn( 2024-06-26T07:18:37.6384147Z 2024-06-26T07:18:49.4440948Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:18:49.4441695Z loading model: 0it [00:11, ?it/s] 2024-06-26T07:18:49.4547094Z cuda eval DebertaV2ForMaskedLM 2024-06-26T07:18:49.4550146Z pass_due_to_skip 2024-06-26T07:18:49.5735492Z TIMING: 2024-06-26T07:18:49.5736116Z STATS: call_* op count: 0 2024-06-26T07:18:49.5737543Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:18:52.8722901Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:18:52.8724803Z warnings.warn( 2024-06-26T07:18:53.0693177Z 2024-06-26T07:19:02.8519271Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:19:02.8520025Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:19:02.8616432Z cuda eval DebertaV2ForQuestionAnswering 2024-06-26T07:20:05.4939509Z pass 2024-06-26T07:20:05.5360463Z TIMING: backend_compile:0.00027 entire_frame_compile:3.6066 2024-06-26T07:20:05.5362778Z STATS: call_* op count: 1086 | FakeTensorMode.__torch_dispatch__:50017 | FakeTensor.__torch_dispatch__:6569 | ProxyTorchDispatchMode.__torch_dispatch__:13673 2024-06-26T07:20:05.5363947Z Dynamo produced 1 graphs covering 1086 ops with 0 graph breaks (0 unique) 2024-06-26T07:20:10.0475888Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:20:10.0477575Z warnings.warn( 2024-06-26T07:20:10.2274449Z 2024-06-26T07:20:11.5365605Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:20:11.5368160Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:20:11.5393015Z cuda eval DistilBertForMaskedLM 2024-06-26T07:20:30.1746252Z pass 2024-06-26T07:20:30.1842913Z TIMING: backend_compile:0.0001 entire_frame_compile:0.7334 2024-06-26T07:20:30.1844016Z STATS: call_* op count: 206 | FakeTensor.__torch_dispatch__:1725 | FakeTensorMode.__torch_dispatch__:13319 | ProxyTorchDispatchMode.__torch_dispatch__:3459 2024-06-26T07:20:30.1845279Z Dynamo produced 1 graphs covering 206 ops with 0 graph breaks (0 unique) 2024-06-26T07:20:33.9904390Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:20:33.9906756Z warnings.warn( 2024-06-26T07:20:34.3362966Z 2024-06-26T07:20:35.4355292Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:20:35.4355746Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:20:35.4381996Z cuda eval DistilBertForQuestionAnswering 2024-06-26T07:20:53.5813566Z pass 2024-06-26T07:20:53.5880912Z TIMING: backend_compile:0.00011 entire_frame_compile:0.74379 2024-06-26T07:20:53.5882093Z STATS: call_* op count: 214 | FakeTensor.__torch_dispatch__:1711 | FakeTensorMode.__torch_dispatch__:13367 | ProxyTorchDispatchMode.__torch_dispatch__:3478 2024-06-26T07:20:53.5883660Z Dynamo produced 1 graphs covering 214 ops with 0 graph breaks (0 unique) 2024-06-26T07:20:57.3890220Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:20:57.5988598Z warnings.warn( 2024-06-26T07:20:57.5988810Z 2024-06-26T07:20:59.3372687Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:20:59.3373173Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:20:59.3394849Z cuda eval DistillGPT2 2024-06-26T07:21:17.9918719Z pass 2024-06-26T07:21:18.0043605Z TIMING: backend_compile:0.00013 entire_frame_compile:1.15005 2024-06-26T07:21:18.0045094Z STATS: call_* op count: 324 | FakeTensorMode.__torch_dispatch__:11826 | FakeTensor.__torch_dispatch__:1270 | ProxyTorchDispatchMode.__torch_dispatch__:3017 2024-06-26T07:21:18.0046228Z Dynamo produced 1 graphs covering 324 ops with 0 graph breaks (0 unique) 2024-06-26T07:21:21.8414198Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:21:21.8415966Z warnings.warn( 2024-06-26T07:21:22.2048146Z 2024-06-26T07:21:22.2057250Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:21:22.7737125Z 2024-06-26T07:21:22.7737324Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:21:22.7795071Z cuda eval ElectraForCausalLM 2024-06-26T07:21:47.9685112Z /tmp/tmpbg5lqifs/chbuunhcbzpz7hhb5mqpyyzxghz7jji7eq6fhnlypoyiyzbaubsg/cfgp7k7z4gwvou2rlu444tpqlz6pyqgbnhwfrii6rtjd7ttugj6s.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:21:47.9688759Z /tmp/tmpbg5lqifs/chbuunhcbzpz7hhb5mqpyyzxghz7jji7eq6fhnlypoyiyzbaubsg/cfgp7k7z4gwvou2rlu444tpqlz6pyqgbnhwfrii6rtjd7ttugj6s.cpp:2653:10: warning: variable ‘electra_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:21:47.9690517Z 2653 | auto electra_embeddings_token_type_ids = constants_->at(204); 2024-06-26T07:21:47.9691109Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:21:48.7357399Z pass 2024-06-26T07:21:48.7498990Z TIMING: backend_compile:0.00014 entire_frame_compile:2.69696 2024-06-26T07:21:48.7500106Z STATS: call_* op count: 375 | FakeTensorMode.__torch_dispatch__:24300 | FakeTensor.__torch_dispatch__:3107 | ProxyTorchDispatchMode.__torch_dispatch__:6846 2024-06-26T07:21:48.7501269Z Dynamo produced 1 graphs covering 375 ops with 0 graph breaks (0 unique) 2024-06-26T07:21:52.7810469Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:21:52.7812275Z warnings.warn( 2024-06-26T07:21:52.9798769Z 2024-06-26T07:21:53.4949637Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:21:53.4950094Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:21:53.5003512Z cuda eval ElectraForQuestionAnswering 2024-06-26T07:22:18.2150271Z /tmp/tmpbg5lqifs/clubqmdzydnmyqegwywl53zb7b52kjnp5y4y5oa5ghfx3p2ac5wg/cwlrkjslctsyjxvkaafpgpmq3xc7ghklgmdpz3kj5v6lysx3p66p.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:22:18.2154281Z /tmp/tmpbg5lqifs/clubqmdzydnmyqegwywl53zb7b52kjnp5y4y5oa5ghfx3p2ac5wg/cwlrkjslctsyjxvkaafpgpmq3xc7ghklgmdpz3kj5v6lysx3p66p.cpp:2616:10: warning: variable ‘electra_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:22:18.2156019Z 2616 | auto electra_embeddings_token_type_ids = constants_->at(200); 2024-06-26T07:22:18.2156612Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:22:18.9684037Z pass 2024-06-26T07:22:18.9762731Z TIMING: backend_compile:0.00014 entire_frame_compile:2.6644 2024-06-26T07:22:18.9763832Z STATS: call_* op count: 378 | FakeTensorMode.__torch_dispatch__:24263 | FakeTensor.__torch_dispatch__:3090 | ProxyTorchDispatchMode.__torch_dispatch__:6840 2024-06-26T07:22:18.9765040Z Dynamo produced 1 graphs covering 378 ops with 0 graph breaks (0 unique) 2024-06-26T07:22:22.9906923Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:22:22.9908973Z warnings.warn( 2024-06-26T07:22:23.2002257Z 2024-06-26T07:22:25.1711527Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:22:25.1712009Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:22:25.1755688Z cuda eval GPT2ForSequenceClassification 2024-06-26T07:22:52.4126473Z pass 2024-06-26T07:22:52.4228221Z TIMING: backend_compile:0.00021 entire_frame_compile:2.2174 2024-06-26T07:22:52.4230254Z STATS: call_* op count: 633 | FakeTensorMode.__torch_dispatch__:22145 | FakeTensor.__torch_dispatch__:2417 | ProxyTorchDispatchMode.__torch_dispatch__:5749 2024-06-26T07:22:52.4231435Z Dynamo produced 1 graphs covering 633 ops with 0 graph breaks (0 unique) 2024-06-26T07:22:56.4208103Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:22:56.4209809Z warnings.warn( 2024-06-26T07:22:56.6839763Z 2024-06-26T07:22:58.0110287Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:22:58.2084596Z 2024-06-26T07:22:58.2085221Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:22:58.2085875Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:22:58.2096871Z cuda eval GoogleFnet 2024-06-26T07:22:58.7363236Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:23:16.4107111Z /tmp/tmpbg5lqifs/cwkxrhhk7gqhu6zwhoatdpsxlrnjmqdzezlubeyqsmzgv6jh54yl/cb4njjg7zllltkbhdr4unaoaavqdlr35payrbkuud3uzibqf6t5m.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:23:16.4110599Z /tmp/tmpbg5lqifs/cwkxrhhk7gqhu6zwhoatdpsxlrnjmqdzezlubeyqsmzgv6jh54yl/cb4njjg7zllltkbhdr4unaoaavqdlr35payrbkuud3uzibqf6t5m.cpp:1695:10: warning: variable ‘fnet_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:23:16.4112335Z 1695 | auto fnet_embeddings_token_type_ids = constants_->at(108); 2024-06-26T07:23:16.4112919Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:23:19.6045515Z pass 2024-06-26T07:23:19.6172322Z TIMING: backend_compile:0.0001 entire_frame_compile:1.14881 2024-06-26T07:23:19.6173438Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:11435 | FakeTensor.__torch_dispatch__:1557 | ProxyTorchDispatchMode.__torch_dispatch__:3016 2024-06-26T07:23:19.6174581Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2024-06-26T07:23:23.4669227Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:23:23.4670936Z warnings.warn( 2024-06-26T07:23:23.6723775Z 2024-06-26T07:23:25.5119878Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:23:25.5120428Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:23:25.5178269Z cuda eval LayoutLMForMaskedLM 2024-06-26T07:23:53.6736357Z pass 2024-06-26T07:23:53.6920482Z TIMING: backend_compile:0.00016 entire_frame_compile:2.76363 2024-06-26T07:23:53.6922770Z STATS: call_* op count: 396 | FakeTensorMode.__torch_dispatch__:24848 | FakeTensor.__torch_dispatch__:3146 | ProxyTorchDispatchMode.__torch_dispatch__:7012 2024-06-26T07:23:53.6923937Z Dynamo produced 1 graphs covering 396 ops with 0 graph breaks (0 unique) 2024-06-26T07:23:57.7128619Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:23:57.7130694Z warnings.warn( 2024-06-26T07:23:57.9284528Z 2024-06-26T07:23:59.5053385Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:23:59.5053843Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:23:59.5110665Z cuda eval LayoutLMForSequenceClassification 2024-06-26T07:24:27.2056647Z pass 2024-06-26T07:24:27.2199209Z TIMING: backend_compile:0.00017 entire_frame_compile:2.72052 2024-06-26T07:24:27.2200624Z STATS: call_* op count: 394 | FakeTensorMode.__torch_dispatch__:24560 | FakeTensor.__torch_dispatch__:3080 | ProxyTorchDispatchMode.__torch_dispatch__:6918 2024-06-26T07:24:27.2203764Z Dynamo produced 1 graphs covering 394 ops with 0 graph breaks (0 unique) 2024-06-26T07:24:31.2407571Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:24:31.2409282Z warnings.warn( 2024-06-26T07:24:31.4462334Z 2024-06-26T07:24:41.2409079Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:24:41.2409552Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:24:41.2528876Z cuda eval M2M100ForConditionalGeneration 2024-06-26T07:24:41.3257557Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:25:46.6345543Z pass 2024-06-26T07:25:46.6671445Z TIMING: backend_compile:0.00029 entire_frame_compile:2.85123 2024-06-26T07:25:46.6672677Z STATS: call_* op count: 1290 | FakeTensorMode.__torch_dispatch__:63968 | FakeTensor.__torch_dispatch__:7808 | ProxyTorchDispatchMode.__torch_dispatch__:17602 2024-06-26T07:25:46.6677055Z Dynamo produced 1 graphs covering 1290 ops with 0 graph breaks (0 unique) 2024-06-26T07:25:51.4000331Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:25:51.4002216Z warnings.warn( 2024-06-26T07:25:51.6049796Z 2024-06-26T07:25:55.2013875Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:25:55.2014350Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:25:55.2094914Z cuda eval MBartForCausalLM 2024-06-26T07:25:55.2389220Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:26:24.6694212Z pass 2024-06-26T07:26:24.6980045Z TIMING: backend_compile:0.00015 entire_frame_compile:1.33757 2024-06-26T07:26:24.6983483Z STATS: call_* op count: 482 | FakeTensorMode.__torch_dispatch__:24652 | FakeTensor.__torch_dispatch__:3204 | ProxyTorchDispatchMode.__torch_dispatch__:6673 2024-06-26T07:26:24.6984639Z Dynamo produced 1 graphs covering 482 ops with 0 graph breaks (0 unique) 2024-06-26T07:26:28.7332447Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:26:28.7334097Z warnings.warn( 2024-06-26T07:26:28.9796679Z 2024-06-26T07:26:35.4282956Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:26:35.4283548Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:26:35.4415268Z cuda eval MBartForConditionalGeneration 2024-06-26T07:26:35.5344614Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:27:39.6260183Z pass 2024-06-26T07:27:39.6601877Z TIMING: backend_compile:0.0003 entire_frame_compile:2.99113 2024-06-26T07:27:39.6604498Z STATS: call_* op count: 1263 | FakeTensorMode.__torch_dispatch__:63856 | FakeTensor.__torch_dispatch__:8269 | ProxyTorchDispatchMode.__torch_dispatch__:17813 2024-06-26T07:27:39.6605930Z Dynamo produced 1 graphs covering 1263 ops with 0 graph breaks (0 unique) 2024-06-26T07:27:44.3497025Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:27:44.3499507Z warnings.warn( 2024-06-26T07:27:44.5722854Z 2024-06-26T07:27:48.3078646Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:27:48.3079297Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:27:48.3134787Z cuda eval MT5ForConditionalGeneration 2024-06-26T07:28:30.3145491Z pass 2024-06-26T07:28:30.3310532Z TIMING: backend_compile:0.0003 entire_frame_compile:4.08827 2024-06-26T07:28:30.3313461Z STATS: call_* op count: 1216 | FakeTensorMode.__torch_dispatch__:41734 | FakeTensor.__torch_dispatch__:4751 | ProxyTorchDispatchMode.__torch_dispatch__:11577 2024-06-26T07:28:30.3314932Z Dynamo produced 1 graphs covering 1216 ops with 0 graph breaks (0 unique) 2024-06-26T07:28:34.5551550Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:28:34.5555249Z warnings.warn( 2024-06-26T07:28:34.9671954Z 2024-06-26T07:28:34.9718219Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:28:39.3326150Z 2024-06-26T07:28:39.3326999Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:28:39.3433456Z cuda eval MegatronBertForCausalLM 2024-06-26T07:29:30.4362302Z pass 2024-06-26T07:29:30.4671508Z TIMING: backend_compile:0.00021 entire_frame_compile:5.05142 2024-06-26T07:29:30.4672623Z STATS: call_* op count: 721 | FakeTensorMode.__torch_dispatch__:46280 | FakeTensor.__torch_dispatch__:5968 | ProxyTorchDispatchMode.__torch_dispatch__:13234 2024-06-26T07:29:30.4674152Z Dynamo produced 1 graphs covering 721 ops with 0 graph breaks (0 unique) 2024-06-26T07:29:34.9356958Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:29:34.9358710Z warnings.warn( 2024-06-26T07:29:35.2000692Z 2024-06-26T07:29:39.3093071Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:29:39.3093558Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:29:39.3202244Z cuda eval MegatronBertForQuestionAnswering 2024-06-26T07:30:29.9609695Z pass 2024-06-26T07:30:29.9871625Z TIMING: backend_compile:0.00021 entire_frame_compile:5.0436 2024-06-26T07:30:29.9874224Z STATS: call_* op count: 724 | FakeTensorMode.__torch_dispatch__:46273 | FakeTensor.__torch_dispatch__:5951 | ProxyTorchDispatchMode.__torch_dispatch__:13228 2024-06-26T07:30:29.9875422Z Dynamo produced 1 graphs covering 724 ops with 0 graph breaks (0 unique) 2024-06-26T07:30:34.4266085Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:30:34.4267737Z warnings.warn( 2024-06-26T07:30:34.8070778Z 2024-06-26T07:30:35.9564372Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:30:35.9565061Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:30:35.9857682Z cuda eval MobileBertForMaskedLM 2024-06-26T07:32:16.1632608Z pass 2024-06-26T07:32:16.1811551Z TIMING: backend_compile:0.00042 entire_frame_compile:13.33122 2024-06-26T07:32:16.1813717Z STATS: call_* op count: 1447 | FakeTensorMode.__torch_dispatch__:92048 | FakeTensor.__torch_dispatch__:10096 | ProxyTorchDispatchMode.__torch_dispatch__:25859 2024-06-26T07:32:16.1815949Z Dynamo produced 1 graphs covering 1447 ops with 0 graph breaks (0 unique) 2024-06-26T07:32:21.2484220Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:21.2486040Z warnings.warn( 2024-06-26T07:32:21.4891471Z 2024-06-26T07:32:22.4792355Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:22.4792834Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:22.5081671Z cuda eval MobileBertForQuestionAnswering 2024-06-26T07:34:00.1637101Z pass 2024-06-26T07:34:00.1724194Z TIMING: backend_compile:0.0004 entire_frame_compile:13.26207 2024-06-26T07:34:00.1727066Z STATS: call_* op count: 1451 | FakeTensorMode.__torch_dispatch__:92178 | FakeTensor.__torch_dispatch__:10092 | ProxyTorchDispatchMode.__torch_dispatch__:25871 2024-06-26T07:34:00.1728328Z Dynamo produced 1 graphs covering 1451 ops with 0 graph breaks (0 unique) 2024-06-26T07:34:05.2279950Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:05.2281639Z warnings.warn( 2024-06-26T07:34:05.4270661Z 2024-06-26T07:34:07.5360136Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:07.5360615Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:34:07.5408845Z cuda eval OPTForCausalLM 2024-06-26T07:34:07.5598259Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-06-26T07:34:35.2349398Z pass 2024-06-26T07:34:35.2533288Z TIMING: backend_compile:0.00016 entire_frame_compile:1.1758 2024-06-26T07:34:35.2535767Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:24418 | FakeTensor.__torch_dispatch__:2974 | ProxyTorchDispatchMode.__torch_dispatch__:6574 2024-06-26T07:34:35.2536958Z Dynamo produced 1 graphs covering 534 ops with 0 graph breaks (0 unique) 2024-06-26T07:34:39.2449091Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:39.2450797Z warnings.warn( 2024-06-26T07:34:39.4677691Z 2024-06-26T07:34:41.1977007Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:41.1977475Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:34:41.2018401Z cuda eval PLBartForCausalLM 2024-06-26T07:34:41.2174226Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:35:00.0078382Z pass 2024-06-26T07:35:00.0226458Z TIMING: backend_compile:0.00011 entire_frame_compile:0.70933 2024-06-26T07:35:00.0227820Z STATS: call_* op count: 253 | FakeTensorMode.__torch_dispatch__:12925 | FakeTensor.__torch_dispatch__:1650 | ProxyTorchDispatchMode.__torch_dispatch__:3456 2024-06-26T07:35:00.0228962Z Dynamo produced 1 graphs covering 253 ops with 0 graph breaks (0 unique) 2024-06-26T07:35:03.8066319Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:35:03.8068008Z warnings.warn( 2024-06-26T07:35:04.0274474Z 2024-06-26T07:35:07.0882930Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:07.0883403Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:35:07.0949478Z cuda eval PLBartForConditionalGeneration 2024-06-26T07:35:07.1426816Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:35:40.2354129Z pass 2024-06-26T07:35:40.2532806Z TIMING: backend_compile:0.00018 entire_frame_compile:1.68182 2024-06-26T07:35:40.2533919Z STATS: call_* op count: 656 | FakeTensorMode.__torch_dispatch__:33047 | FakeTensor.__torch_dispatch__:4209 | ProxyTorchDispatchMode.__torch_dispatch__:9116 2024-06-26T07:35:40.2535047Z Dynamo produced 1 graphs covering 656 ops with 0 graph breaks (0 unique) 2024-06-26T07:35:44.4153594Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:35:44.4155446Z warnings.warn( 2024-06-26T07:35:44.5991065Z 2024-06-26T07:35:50.5888034Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:50.5888601Z loading model: 0it [00:05, ?it/s] 2024-06-26T07:35:50.5968147Z cuda eval PegasusForCausalLM 2024-06-26T07:35:50.6272086Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-06-26T07:36:20.1385143Z pass 2024-06-26T07:36:20.1616793Z TIMING: backend_compile:0.00015 entire_frame_compile:1.32651 2024-06-26T07:36:20.1617945Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:24453 | FakeTensor.__torch_dispatch__:3166 | ProxyTorchDispatchMode.__torch_dispatch__:6620 2024-06-26T07:36:20.1619889Z Dynamo produced 1 graphs covering 478 ops with 0 graph breaks (0 unique) 2024-06-26T07:36:24.1842329Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:24.1844062Z warnings.warn( 2024-06-26T07:36:24.3955556Z 2024-06-26T07:36:34.5568417Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:34.5568925Z loading model: 0it [00:10, ?it/s] 2024-06-26T07:36:34.5699824Z cuda eval PegasusForConditionalGeneration 2024-06-26T07:36:34.6423516Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:37:37.2246768Z pass 2024-06-26T07:37:37.2492453Z TIMING: backend_compile:0.00029 entire_frame_compile:2.97696 2024-06-26T07:37:37.2494455Z STATS: call_* op count: 1242 | FakeTensorMode.__torch_dispatch__:63220 | FakeTensor.__torch_dispatch__:8163 | ProxyTorchDispatchMode.__torch_dispatch__:17600 2024-06-26T07:37:37.2495655Z Dynamo produced 1 graphs covering 1242 ops with 0 graph breaks (0 unique) 2024-06-26T07:37:41.9232598Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:41.9234955Z warnings.warn( 2024-06-26T07:37:42.1523988Z 2024-06-26T07:37:42.1571688Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-06-26T07:37:44.1199932Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:37:44.1201946Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:37:44.2898251Z 2024-06-26T07:37:44.2898826Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:37:44.2955174Z cuda eval RobertaForCausalLM 2024-06-26T07:38:09.7700277Z /tmp/tmpbg5lqifs/cnrdr3s4qu64zonktracdj4cqhdtamzlhfmhseczls5wd7kdhze3/chfpfbbzucbtjlo5poy6fpyzjeci63litdcbgyqdigoxshdqxm2v.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:38:09.7703840Z /tmp/tmpbg5lqifs/cnrdr3s4qu64zonktracdj4cqhdtamzlhfmhseczls5wd7kdhze3/chfpfbbzucbtjlo5poy6fpyzjeci63litdcbgyqdigoxshdqxm2v.cpp:2625:10: warning: variable ‘roberta_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:38:09.7705567Z 2625 | auto roberta_embeddings_token_type_ids = constants_->at(202); 2024-06-26T07:38:09.7706168Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:38:12.2777064Z pass 2024-06-26T07:38:12.2944903Z TIMING: backend_compile:0.00014 entire_frame_compile:2.66441 2024-06-26T07:38:12.2947096Z STATS: call_* op count: 382 | FakeTensorMode.__torch_dispatch__:24179 | FakeTensor.__torch_dispatch__:3112 | ProxyTorchDispatchMode.__torch_dispatch__:6849 2024-06-26T07:38:12.2948672Z Dynamo produced 1 graphs covering 382 ops with 0 graph breaks (0 unique) 2024-06-26T07:38:16.3508609Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:16.3510332Z warnings.warn( 2024-06-26T07:38:16.5557597Z 2024-06-26T07:38:18.1048973Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:38:18.1051067Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:38:18.2879540Z 2024-06-26T07:38:18.2879754Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:38:18.2935254Z cuda eval RobertaForQuestionAnswering 2024-06-26T07:38:43.3077900Z /tmp/tmpbg5lqifs/ceue4esqvvaysdh4eo6apzvaghegj3ra7l5y4ptzp56tiqaqf5vu/chdoh3plgol3qsven3lpxgoa4epcix6jl5lq4lm3xnpjnl6z6jtd.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:38:43.3081058Z /tmp/tmpbg5lqifs/ceue4esqvvaysdh4eo6apzvaghegj3ra7l5y4ptzp56tiqaqf5vu/chdoh3plgol3qsven3lpxgoa4epcix6jl5lq4lm3xnpjnl6z6jtd.cpp:2588:10: warning: variable ‘roberta_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:38:43.3082787Z 2588 | auto roberta_embeddings_token_type_ids = constants_->at(198); 2024-06-26T07:38:43.3083384Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:38:45.7685422Z pass 2024-06-26T07:38:45.7772384Z TIMING: backend_compile:0.00014 entire_frame_compile:2.65833 2024-06-26T07:38:45.7773584Z STATS: call_* op count: 384 | FakeTensorMode.__torch_dispatch__:24161 | FakeTensor.__torch_dispatch__:3095 | ProxyTorchDispatchMode.__torch_dispatch__:6837 2024-06-26T07:38:45.7774749Z Dynamo produced 1 graphs covering 384 ops with 0 graph breaks (0 unique) 2024-06-26T07:38:49.8257594Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:49.8259305Z warnings.warn( 2024-06-26T07:38:50.0885533Z 2024-06-26T07:38:50.6118873Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:50.6119370Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:50.6161126Z cuda eval Speech2Text2ForCausalLM 2024-06-26T07:38:50.6318819Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-06-26T07:39:07.6810709Z pass 2024-06-26T07:39:07.6828692Z TIMING: backend_compile:0.00011 entire_frame_compile:0.67555 2024-06-26T07:39:07.6829794Z STATS: call_* op count: 259 | FakeTensorMode.__torch_dispatch__:12792 | FakeTensor.__torch_dispatch__:1541 | ProxyTorchDispatchMode.__torch_dispatch__:3397 2024-06-26T07:39:07.6831217Z Dynamo produced 1 graphs covering 259 ops with 0 graph breaks (0 unique) 2024-06-26T07:39:11.4525730Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:11.4527975Z warnings.warn( 2024-06-26T07:39:11.6947142Z 2024-06-26T07:39:12.8825258Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:12.8825895Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:12.8866080Z cuda eval T5ForConditionalGeneration 2024-06-26T07:39:43.9995046Z pass 2024-06-26T07:39:44.0380254Z TIMING: backend_compile:0.00022 entire_frame_compile:2.84986 2024-06-26T07:39:44.0382445Z STATS: call_* op count: 831 | FakeTensorMode.__torch_dispatch__:29153 | FakeTensor.__torch_dispatch__:3249 | ProxyTorchDispatchMode.__torch_dispatch__:8034 2024-06-26T07:39:44.0384295Z Dynamo produced 1 graphs covering 831 ops with 0 graph breaks (0 unique) 2024-06-26T07:39:48.0650484Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:48.0652092Z warnings.warn( 2024-06-26T07:39:48.2720259Z 2024-06-26T07:39:49.4493401Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:49.4493955Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:49.4532945Z cuda eval T5Small 2024-06-26T07:40:18.9120719Z pass 2024-06-26T07:40:18.9503450Z TIMING: backend_compile:0.00023 entire_frame_compile:2.83023 2024-06-26T07:40:18.9505984Z STATS: call_* op count: 831 | FakeTensorMode.__torch_dispatch__:29153 | FakeTensor.__torch_dispatch__:3249 | ProxyTorchDispatchMode.__torch_dispatch__:8034 2024-06-26T07:40:18.9507167Z Dynamo produced 1 graphs covering 831 ops with 0 graph breaks (0 unique) 2024-06-26T07:40:22.9816128Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:22.9817810Z warnings.warn( 2024-06-26T07:40:23.2156968Z 2024-06-26T07:40:26.7861054Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:26.7862148Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:40:26.7941466Z cuda eval TrOCRForCausalLM 2024-06-26T07:40:26.8255141Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-06-26T07:40:55.8304356Z pass 2024-06-26T07:40:55.8568805Z TIMING: backend_compile:0.00015 entire_frame_compile:1.34426 2024-06-26T07:40:55.8571148Z STATS: call_* op count: 479 | FakeTensorMode.__torch_dispatch__:24494 | FakeTensor.__torch_dispatch__:3170 | ProxyTorchDispatchMode.__torch_dispatch__:6624 2024-06-26T07:40:55.8572508Z Dynamo produced 1 graphs covering 479 ops with 0 graph breaks (0 unique) 2024-06-26T07:40:59.8268312Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:59.8269968Z warnings.warn( 2024-06-26T07:41:00.0387873Z 2024-06-26T07:41:09.1039229Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:41:09.1039784Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:41:09.1140920Z cuda eval XGLMForCausalLM 2024-06-26T07:41:09.2594147Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-06-26T07:42:04.2630558Z pass 2024-06-26T07:42:04.2967340Z TIMING: backend_compile:0.00024 entire_frame_compile:2.37838 2024-06-26T07:42:04.2969393Z STATS: call_* op count: 991 | FakeTensorMode.__torch_dispatch__:48842 | FakeTensor.__torch_dispatch__:6280 | ProxyTorchDispatchMode.__torch_dispatch__:13308 2024-06-26T07:42:04.2970564Z Dynamo produced 1 graphs covering 991 ops with 0 graph breaks (0 unique) 2024-06-26T07:42:08.9081836Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:42:08.9083515Z warnings.warn( 2024-06-26T07:42:09.1684927Z 2024-06-26T07:42:13.1613442Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:13.1613927Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:42:13.1708384Z cuda eval XLNetLMHeadModel 2024-06-26T07:42:41.1667548Z W0626 07:42:41.166000 140571891491456 torch/_inductor/utils.py:1234] DeviceCopy in input program 2024-06-26T07:43:21.1673462Z pass 2024-06-26T07:43:21.2061066Z TIMING: backend_compile:0.00025 entire_frame_compile:5.37579 2024-06-26T07:43:21.2063715Z STATS: call_* op count: 816 | FakeTensorMode.__torch_dispatch__:79102 | FakeTensor.__torch_dispatch__:8021 | ProxyTorchDispatchMode.__torch_dispatch__:20588 2024-06-26T07:43:21.2064903Z Dynamo produced 1 graphs covering 816 ops with 0 graph breaks (0 unique) 2024-06-26T07:43:25.9739204Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:43:25.9740895Z warnings.warn( 2024-06-26T07:43:26.2378608Z 2024-06-26T07:43:28.2177204Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:43:28.2177800Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:43:28.2251948Z cuda eval YituTechConvBert 2024-06-26T07:44:03.9555265Z /tmp/tmpbg5lqifs/cpp4jno56noobykjfgfi5lbxurhvgczorj5sp5qrhm7mqlhvp3zb/cp4j6hm6c7czqseub3mgxgbqmmbbolt7vke7cjwddoqnadn26x42.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-06-26T07:44:03.9558755Z /tmp/tmpbg5lqifs/cpp4jno56noobykjfgfi5lbxurhvgczorj5sp5qrhm7mqlhvp3zb/cp4j6hm6c7czqseub3mgxgbqmmbbolt7vke7cjwddoqnadn26x42.cpp:3477:10: warning: variable ‘convbert_embeddings_token_type_ids’ set but not used [-Wunused-but-set-variable] 2024-06-26T07:44:03.9560482Z 3477 | auto convbert_embeddings_token_type_ids = constants_->at(286); 2024-06-26T07:44:03.9561089Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-06-26T07:44:06.2117009Z pass 2024-06-26T07:44:06.2316808Z TIMING: backend_compile:0.00019 entire_frame_compile:3.69057 2024-06-26T07:44:06.2318618Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:37061 | FakeTensor.__torch_dispatch__:4541 | ProxyTorchDispatchMode.__torch_dispatch__:9947 2024-06-26T07:44:06.2319934Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2024-06-26T07:44:07.8532189Z accuracy pass_rate=93.48% 2024-06-26T07:44:07.8536065Z calls_captured gmean=0.00x mean=599.391x 2024-06-26T07:44:07.8540221Z unique_graphs gmean=0.00x mean=0.935x 2024-06-26T07:44:07.8544255Z graph_breaks gmean=0.00x mean=0.000x 2024-06-26T07:44:07.8547924Z unique_graph_breaks gmean=0.00x mean=0.000x 2024-06-26T07:44:07.8551788Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T07:44:07.8556291Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T07:44:07.8560050Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T07:44:10.8577155Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_inductor_huggingface_inference.csv 2024-06-26T07:44:11.0974546Z AlbertForMaskedLM PASS 2024-06-26T07:44:11.0978929Z AlbertForQuestionAnswering PASS 2024-06-26T07:44:11.0983393Z AllenaiLongformerBase XFAIL 2024-06-26T07:44:11.0987903Z BartForCausalLM PASS 2024-06-26T07:44:11.0992369Z BartForConditionalGeneration PASS 2024-06-26T07:44:11.0996926Z BertForMaskedLM PASS 2024-06-26T07:44:11.1001277Z BertForQuestionAnswering PASS 2024-06-26T07:44:11.1006142Z BlenderbotForCausalLM XFAIL 2024-06-26T07:44:11.1010397Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:44:11.1014818Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:44:11.1019200Z CamemBert PASS 2024-06-26T07:44:11.1023720Z DebertaForMaskedLM PASS 2024-06-26T07:44:11.1028128Z DebertaForQuestionAnswering PASS 2024-06-26T07:44:11.1032538Z DebertaV2ForMaskedLM XFAIL 2024-06-26T07:44:11.1037091Z DebertaV2ForQuestionAnswering PASS 2024-06-26T07:44:11.1041474Z DistilBertForMaskedLM PASS 2024-06-26T07:44:11.1046217Z DistilBertForQuestionAnswering PASS 2024-06-26T07:44:11.1050510Z DistillGPT2 PASS 2024-06-26T07:44:11.1054892Z ElectraForCausalLM PASS 2024-06-26T07:44:11.1059266Z ElectraForQuestionAnswering PASS 2024-06-26T07:44:11.1063751Z GPT2ForSequenceClassification PASS 2024-06-26T07:44:11.1068148Z GoogleFnet PASS 2024-06-26T07:44:11.1072525Z LayoutLMForMaskedLM PASS 2024-06-26T07:44:11.1077531Z LayoutLMForSequenceClassification PASS 2024-06-26T07:44:11.1081981Z M2M100ForConditionalGeneration PASS 2024-06-26T07:44:11.1086579Z MBartForCausalLM PASS 2024-06-26T07:44:11.1091003Z MBartForConditionalGeneration PASS 2024-06-26T07:44:11.1095399Z MT5ForConditionalGeneration PASS 2024-06-26T07:44:11.1099752Z MegatronBertForCausalLM PASS 2024-06-26T07:44:11.1104168Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:44:11.1108575Z MobileBertForMaskedLM PASS 2024-06-26T07:44:11.1112945Z MobileBertForQuestionAnswering PASS 2024-06-26T07:44:11.1117537Z OPTForCausalLM PASS 2024-06-26T07:44:11.1121890Z PLBartForCausalLM PASS 2024-06-26T07:44:11.1126617Z PLBartForConditionalGeneration PASS 2024-06-26T07:44:11.1130932Z PegasusForCausalLM PASS 2024-06-26T07:44:11.1135302Z PegasusForConditionalGeneration PASS 2024-06-26T07:44:11.1139704Z RobertaForCausalLM PASS 2024-06-26T07:44:11.1144086Z RobertaForQuestionAnswering PASS 2024-06-26T07:44:11.1148493Z Speech2Text2ForCausalLM PASS 2024-06-26T07:44:11.1152919Z T5ForConditionalGeneration PASS 2024-06-26T07:44:11.1157567Z T5Small PASS 2024-06-26T07:44:11.1161939Z TrOCRForCausalLM PASS 2024-06-26T07:44:11.1166649Z XGLMForCausalLM PASS 2024-06-26T07:44:11.1170833Z XLNetLMHeadModel PASS 2024-06-26T07:44:11.1175164Z YituTechConvBert PASS 2024-06-26T07:44:11.1560643Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_inductor_huggingface_inference.csv 2024-06-26T07:44:11.4253169Z AlbertForMaskedLM PASS 2024-06-26T07:44:11.4256588Z AlbertForQuestionAnswering PASS 2024-06-26T07:44:11.4261306Z AllenaiLongformerBase PASS 2024-06-26T07:44:11.4266297Z BartForCausalLM PASS 2024-06-26T07:44:11.4271156Z BartForConditionalGeneration PASS 2024-06-26T07:44:11.4276619Z BertForMaskedLM PASS 2024-06-26T07:44:11.4280904Z BertForQuestionAnswering PASS 2024-06-26T07:44:11.4286108Z BlenderbotForCausalLM PASS 2024-06-26T07:44:11.4290971Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:44:11.4295746Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:44:11.4300416Z CamemBert PASS 2024-06-26T07:44:11.4305183Z DebertaForMaskedLM PASS 2024-06-26T07:44:11.4310025Z DebertaForQuestionAnswering PASS 2024-06-26T07:44:11.4315068Z DebertaV2ForMaskedLM PASS 2024-06-26T07:44:11.4319736Z DebertaV2ForQuestionAnswering PASS 2024-06-26T07:44:11.4324845Z DistilBertForMaskedLM PASS 2024-06-26T07:44:11.4329972Z DistilBertForQuestionAnswering PASS 2024-06-26T07:44:11.4334558Z DistillGPT2 PASS 2024-06-26T07:44:11.4339534Z ElectraForCausalLM PASS 2024-06-26T07:44:11.4344558Z ElectraForQuestionAnswering PASS 2024-06-26T07:44:11.4349447Z GPT2ForSequenceClassification PASS 2024-06-26T07:44:11.4354541Z GoogleFnet PASS 2024-06-26T07:44:11.4359423Z LayoutLMForMaskedLM PASS 2024-06-26T07:44:11.4364190Z LayoutLMForSequenceClassification PASS 2024-06-26T07:44:11.4369394Z M2M100ForConditionalGeneration PASS 2024-06-26T07:44:11.4374289Z MBartForCausalLM PASS 2024-06-26T07:44:11.4379171Z MBartForConditionalGeneration PASS 2024-06-26T07:44:11.4384048Z MT5ForConditionalGeneration PASS 2024-06-26T07:44:11.4388847Z MegatronBertForCausalLM PASS 2024-06-26T07:44:11.4394009Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:44:11.4398958Z MobileBertForMaskedLM PASS 2024-06-26T07:44:11.4403924Z MobileBertForQuestionAnswering PASS 2024-06-26T07:44:11.4409172Z OPTForCausalLM PASS 2024-06-26T07:44:11.4413864Z PLBartForCausalLM PASS 2024-06-26T07:44:11.4418685Z PLBartForConditionalGeneration PASS 2024-06-26T07:44:11.4423588Z PegasusForCausalLM PASS 2024-06-26T07:44:11.4428483Z PegasusForConditionalGeneration PASS 2024-06-26T07:44:11.4433339Z RobertaForCausalLM PASS 2024-06-26T07:44:11.4438493Z RobertaForQuestionAnswering PASS 2024-06-26T07:44:11.4443450Z Speech2Text2ForCausalLM PASS 2024-06-26T07:44:11.4448544Z T5ForConditionalGeneration PASS 2024-06-26T07:44:11.4453308Z T5Small PASS 2024-06-26T07:44:11.4458202Z TrOCRForCausalLM PASS 2024-06-26T07:44:11.4463040Z XGLMForCausalLM PASS 2024-06-26T07:44:11.4468024Z XLNetLMHeadModel PASS 2024-06-26T07:44:11.4472906Z YituTechConvBert PASS 2024-06-26T07:44:11.4870317Z + cleanup_workspace 2024-06-26T07:44:11.4871384Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T07:44:11.4872660Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T07:44:11.4873645Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T07:44:11.4874588Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T07:44:11.4875444Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T07:44:11.4876334Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T07:44:11.4877038Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T07:44:12.0988630Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T07:44:12.0989149Z cat test/**/*_toprint.log || true 2024-06-26T07:44:12.0997411Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:12.0997910Z env: 2024-06-26T07:44:12.0998179Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.0998634Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.0999360Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.0999987Z ##[endgroup] 2024-06-26T07:44:12.1062028Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T07:44:12.1089526Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:44:12.1090139Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:44:12.1097060Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:12.1097592Z env: 2024-06-26T07:44:12.1097877Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.1098351Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.1099136Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.1099866Z MONITOR_SCRIPT_PID: 23279 2024-06-26T07:44:12.1100256Z ##[endgroup] 2024-06-26T07:44:12.1286223Z Prepare all required actions 2024-06-26T07:44:12.1286676Z Getting action download info 2024-06-26T07:44:12.2989474Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T07:44:12.4665483Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T07:44:12.4665964Z with: 2024-06-26T07:44:12.4666530Z file-suffix: test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257 2024-06-26T07:44:12.4667238Z s3-bucket: gha-artifacts 2024-06-26T07:44:12.4667579Z env: 2024-06-26T07:44:12.4667864Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.4668324Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.4669055Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.4669681Z ##[endgroup] 2024-06-26T07:44:12.4694799Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T07:44:12.4695463Z # Remove any previous test jsons if they exist 2024-06-26T07:44:12.4695984Z rm -f test-jsons-*.zip 2024-06-26T07:44:12.4696504Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T07:44:12.4704122Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:12.4704628Z env: 2024-06-26T07:44:12.4704911Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.4705362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.4706091Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.4706999Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257 2024-06-26T07:44:12.4707670Z ##[endgroup] 2024-06-26T07:44:12.4894773Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T07:44:12.4923847Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T07:44:12.4924662Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T07:44:12.4930611Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T07:44:12.4956816Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T07:44:12.4957461Z # Remove any previous test reports if they exist 2024-06-26T07:44:12.4957994Z rm -f test-reports-*.zip 2024-06-26T07:44:12.4958587Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T07:44:12.4966346Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:12.4966841Z env: 2024-06-26T07:44:12.4967127Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.4967583Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.4968309Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.4969213Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257 2024-06-26T07:44:12.4969883Z ##[endgroup] 2024-06-26T07:44:12.5122318Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-06-26T07:44:12.5147580Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T07:44:12.5148227Z # Remove any previous usage logs if they exist 2024-06-26T07:44:12.5148751Z rm -f logs-*.zip 2024-06-26T07:44:12.5149440Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T07:44:12.5150256Z # so check to see if the file exists first 2024-06-26T07:44:12.5150892Z if [ -f 'usage_log.txt' ]; then 2024-06-26T07:44:12.5151442Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T07:44:12.5151943Z fi 2024-06-26T07:44:12.5152324Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T07:44:12.5152914Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T07:44:12.5153423Z fi 2024-06-26T07:44:12.5160932Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:12.5161541Z env: 2024-06-26T07:44:12.5161820Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.5162286Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.5163014Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.5163925Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257 2024-06-26T07:44:12.5164710Z ##[endgroup] 2024-06-26T07:44:12.5214476Z adding: usage_log.txt (deflated 92%) 2024-06-26T07:44:12.5279264Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T07:44:12.5280052Z # Remove any previous debugging artifacts if they exist 2024-06-26T07:44:12.5280612Z rm -f debug-*.zip 2024-06-26T07:44:12.5281001Z if [ -d 'test/debug' ]; then 2024-06-26T07:44:12.5281508Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T07:44:12.5281989Z fi 2024-06-26T07:44:12.5289699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:12.5290250Z env: 2024-06-26T07:44:12.5290527Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.5290984Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.5291706Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.5292610Z FILE_SUFFIX: test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257 2024-06-26T07:44:12.5293285Z ##[endgroup] 2024-06-26T07:44:12.5366748Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:44:12.5367181Z with: 2024-06-26T07:44:12.5367464Z s3-bucket: gha-artifacts 2024-06-26T07:44:12.5367893Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T07:44:12.5368360Z retention-days: 14 2024-06-26T07:44:12.5368693Z if-no-files-found: warn 2024-06-26T07:44:12.5369052Z path: test-jsons-*.zip 2024-06-26T07:44:12.5369394Z name: artifact 2024-06-26T07:44:12.5369711Z region: us-east-1 2024-06-26T07:44:12.5370039Z env: 2024-06-26T07:44:12.5370328Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.5370784Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.5371498Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.5372128Z ##[endgroup] 2024-06-26T07:44:12.8491665Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:44:12.8492390Z With the provided path, there will be 1 file uploaded 2024-06-26T07:44:12.8493024Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T07:44:12.8518721Z Starting upload of test-jsons-test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257.zip 2024-06-26T07:44:12.9592483Z Finished upload of test-jsons-test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257.zip 2024-06-26T07:44:12.9739305Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:44:12.9739744Z with: 2024-06-26T07:44:12.9740029Z s3-bucket: gha-artifacts 2024-06-26T07:44:12.9740471Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T07:44:12.9740941Z retention-days: 14 2024-06-26T07:44:12.9741279Z if-no-files-found: error 2024-06-26T07:44:12.9741646Z path: test-reports-*.zip 2024-06-26T07:44:12.9741998Z name: artifact 2024-06-26T07:44:12.9742295Z region: us-east-1 2024-06-26T07:44:12.9742598Z env: 2024-06-26T07:44:12.9742874Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:12.9743557Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:12.9744427Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:12.9745058Z ##[endgroup] 2024-06-26T07:44:13.2887446Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:44:13.2888260Z With the provided path, there will be 1 file uploaded 2024-06-26T07:44:13.2888956Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T07:44:13.2914458Z Starting upload of test-reports-test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257.zip 2024-06-26T07:44:13.3868295Z Finished upload of test-reports-test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257.zip 2024-06-26T07:44:13.4013812Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:44:13.4014259Z with: 2024-06-26T07:44:13.4014543Z s3-bucket: gha-artifacts 2024-06-26T07:44:13.4014972Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T07:44:13.4015446Z retention-days: 14 2024-06-26T07:44:13.4015797Z if-no-files-found: ignore 2024-06-26T07:44:13.4016158Z path: logs-*.zip 2024-06-26T07:44:13.4016478Z name: artifact 2024-06-26T07:44:13.4016776Z region: us-east-1 2024-06-26T07:44:13.4017080Z env: 2024-06-26T07:44:13.4017357Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:13.4017818Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:13.4018541Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:13.4019172Z ##[endgroup] 2024-06-26T07:44:13.7121591Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:44:13.7122215Z With the provided path, there will be 1 file uploaded 2024-06-26T07:44:13.7122836Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T07:44:13.7148383Z Starting upload of logs-test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257.zip 2024-06-26T07:44:13.8587079Z Finished upload of logs-test-aot_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688912257.zip 2024-06-26T07:44:13.8730597Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:44:13.8731039Z with: 2024-06-26T07:44:13.8731323Z s3-bucket: gha-artifacts 2024-06-26T07:44:13.8731751Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T07:44:13.8732220Z retention-days: 14 2024-06-26T07:44:13.8732550Z if-no-files-found: ignore 2024-06-26T07:44:13.8732915Z path: debug-*.zip 2024-06-26T07:44:13.8733234Z name: artifact 2024-06-26T07:44:13.8733532Z region: us-east-1 2024-06-26T07:44:13.8733831Z env: 2024-06-26T07:44:13.8734119Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:13.8734574Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:13.8735294Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:13.8735917Z ##[endgroup] 2024-06-26T07:44:14.1791202Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T07:44:14.1934073Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T07:44:14.1934580Z # shellcheck disable=SC2156 2024-06-26T07:44:14.1935382Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T07:44:14.1943449Z shell: /usr/bin/bash -e {0} 2024-06-26T07:44:14.1943805Z env: 2024-06-26T07:44:14.1944088Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:14.1944553Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:14.1945291Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:14.1945923Z ##[endgroup] 2024-06-26T07:44:14.3564715Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T07:44:14.3565295Z with: 2024-06-26T07:44:14.3565556Z env: 2024-06-26T07:44:14.3565833Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:14.3566289Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:14.3567028Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:14.3567784Z ##[endgroup] 2024-06-26T07:44:14.3586440Z ##[group]Run set -eou pipefail 2024-06-26T07:44:14.3587003Z set -eou pipefail 2024-06-26T07:44:14.3587447Z  2024-06-26T07:44:14.3587965Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T07:44:14.3588601Z for _ in $(seq 1440); do 2024-06-26T07:44:14.3589079Z  # Break if no ssh session exists anymore 2024-06-26T07:44:14.3589581Z  if [ "$(who)" = "" ]; then 2024-06-26T07:44:14.3589989Z  break 2024-06-26T07:44:14.3590327Z  fi 2024-06-26T07:44:14.3590626Z  echo "." 2024-06-26T07:44:14.3590959Z  sleep 5 2024-06-26T07:44:14.3591287Z done 2024-06-26T07:44:14.3599338Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:14.3599836Z env: 2024-06-26T07:44:14.3600112Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:14.3600604Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:14.3601342Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:14.3601973Z ##[endgroup] 2024-06-26T07:44:14.3626870Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T07:44:14.3663762Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:44:14.3664527Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:44:14.3665123Z # shellcheck disable=SC2046 2024-06-26T07:44:14.3665597Z docker stop $(docker ps -q) || true 2024-06-26T07:44:14.3666072Z # Prune all of the docker images 2024-06-26T07:44:14.3666528Z docker system prune -af 2024-06-26T07:44:14.3673406Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:14.3674155Z env: 2024-06-26T07:44:14.3674433Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:44:14.3674892Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:44:14.3675623Z DOCKER_CONTAINER_ID: 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:14.3676250Z ##[endgroup] 2024-06-26T07:44:14.6746415Z 6b4591b4f61d 2024-06-26T07:44:15.0547493Z Deleted Containers: 2024-06-26T07:44:15.0547998Z 6b4591b4f61d1e59a12061031510473a7d4533b29b305faad91514b17e7515b5 2024-06-26T07:44:15.0548476Z 2024-06-26T07:44:18.6235792Z Deleted Images: 2024-06-26T07:44:18.6237481Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:44:18.6240433Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:44:18.6242085Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T07:44:18.6242936Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T07:44:18.6243844Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T07:44:18.6244834Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T07:44:18.6245705Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T07:44:18.6246564Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T07:44:18.6247395Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T07:44:18.6248482Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T07:44:18.6249341Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T07:44:18.6250167Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T07:44:18.6250993Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T07:44:18.6252007Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T07:44:18.6252849Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T07:44:18.6253694Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T07:44:18.6254536Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T07:44:18.6255379Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T07:44:18.6256234Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T07:44:18.6257083Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T07:44:18.6257934Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T07:44:18.6258788Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T07:44:18.6259638Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T07:44:18.6260474Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T07:44:18.6261298Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T07:44:18.6262143Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T07:44:18.6263011Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T07:44:18.6263885Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T07:44:18.6264756Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T07:44:18.6265624Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T07:44:18.6266484Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T07:44:18.6267320Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T07:44:18.6268155Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T07:44:18.6269000Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T07:44:18.6269857Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T07:44:18.6270701Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T07:44:18.6271555Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T07:44:18.6272456Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T07:44:18.6273301Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T07:44:18.6274324Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T07:44:18.6275163Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T07:44:18.6276000Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T07:44:18.6276927Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T07:44:18.6277921Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T07:44:18.6278785Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T07:44:18.6279623Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T07:44:18.6280597Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T07:44:18.6281466Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T07:44:18.6282306Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T07:44:18.6283170Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T07:44:18.6284072Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T07:44:18.6285039Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T07:44:18.6287416Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T07:44:18.6288260Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T07:44:18.6289094Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T07:44:18.6289939Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T07:44:18.6290787Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T07:44:18.6291617Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T07:44:18.6292462Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T07:44:18.6293291Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T07:44:18.6294128Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T07:44:18.6294961Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T07:44:18.6295803Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T07:44:18.6296635Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T07:44:18.6297473Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T07:44:18.6298315Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T07:44:18.6299148Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T07:44:18.6299965Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T07:44:18.6300805Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T07:44:18.6301689Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T07:44:18.6302518Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T07:44:18.6303345Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T07:44:18.6304165Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T07:44:18.6304996Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T07:44:18.6305846Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T07:44:18.6306667Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T07:44:18.6307514Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T07:44:18.6308363Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T07:44:18.6309226Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T07:44:18.6310074Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T07:44:18.6310893Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T07:44:18.6311397Z 2024-06-26T07:44:18.6311542Z Total reclaimed space: 31.35GB 2024-06-26T07:44:18.6374800Z Post job cleanup. 2024-06-26T07:44:18.6422928Z Post job cleanup. 2024-06-26T07:44:18.7245706Z [command]/usr/bin/git version 2024-06-26T07:44:18.7287237Z git version 2.40.1 2024-06-26T07:44:18.7326947Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/bb5f2dfb-fa55-4a66-afcb-6fbbcce30242' before making global git config changes 2024-06-26T07:44:18.7328223Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:44:18.7332654Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:44:18.7364720Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:44:18.7387453Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:44:18.7602468Z Entering 'android/libs/fbjni' 2024-06-26T07:44:18.7639842Z Entering 'third_party/FP16' 2024-06-26T07:44:18.7678249Z Entering 'third_party/FXdiv' 2024-06-26T07:44:18.7713640Z Entering 'third_party/NNPACK' 2024-06-26T07:44:18.7751949Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:44:18.7789603Z Entering 'third_party/XNNPACK' 2024-06-26T07:44:18.7835867Z Entering 'third_party/benchmark' 2024-06-26T07:44:18.7872662Z Entering 'third_party/cpp-httplib' 2024-06-26T07:44:18.7908289Z Entering 'third_party/cpuinfo' 2024-06-26T07:44:18.7945604Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:44:18.7980645Z Entering 'third_party/cutlass' 2024-06-26T07:44:18.8022524Z Entering 'third_party/eigen' 2024-06-26T07:44:18.8058883Z Entering 'third_party/fbgemm' 2024-06-26T07:44:18.8094200Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:44:18.8128901Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:44:18.8163767Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:44:18.8200680Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:44:18.8233311Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:44:18.8267313Z Entering 'third_party/flatbuffers' 2024-06-26T07:44:18.8305789Z Entering 'third_party/fmt' 2024-06-26T07:44:18.8340446Z Entering 'third_party/foxi' 2024-06-26T07:44:18.8375267Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:44:18.8408206Z Entering 'third_party/gloo' 2024-06-26T07:44:18.8442669Z Entering 'third_party/googletest' 2024-06-26T07:44:18.8476917Z Entering 'third_party/ideep' 2024-06-26T07:44:18.8511125Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:44:18.8551201Z Entering 'third_party/ittapi' 2024-06-26T07:44:18.8587108Z Entering 'third_party/kineto' 2024-06-26T07:44:18.8622764Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:44:18.8659478Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:44:18.8694160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:44:18.8729107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:44:18.8762917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:44:18.8796391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:44:18.8830785Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:44:18.8864884Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:44:18.8900551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:44:18.8935653Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:44:18.8973266Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:44:18.9009578Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:44:18.9044156Z Entering 'third_party/mimalloc' 2024-06-26T07:44:18.9079242Z Entering 'third_party/nccl/nccl' 2024-06-26T07:44:18.9115119Z Entering 'third_party/nlohmann' 2024-06-26T07:44:18.9149642Z Entering 'third_party/onnx' 2024-06-26T07:44:18.9197843Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:44:18.9231732Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:44:18.9269480Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:44:18.9307020Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:44:18.9343221Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:44:18.9378301Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:44:18.9411199Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:44:18.9449717Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:44:18.9483136Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:44:18.9516830Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:44:18.9550001Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:44:18.9586112Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:44:18.9620741Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:44:18.9669408Z Entering 'third_party/pocketfft' 2024-06-26T07:44:18.9703479Z Entering 'third_party/protobuf' 2024-06-26T07:44:18.9740893Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:44:18.9773572Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:44:18.9808738Z Entering 'third_party/psimd' 2024-06-26T07:44:18.9843811Z Entering 'third_party/pthreadpool' 2024-06-26T07:44:18.9878854Z Entering 'third_party/pybind11' 2024-06-26T07:44:18.9912866Z Entering 'third_party/python-peachpy' 2024-06-26T07:44:18.9947561Z Entering 'third_party/sleef' 2024-06-26T07:44:18.9983675Z Entering 'third_party/tensorpipe' 2024-06-26T07:44:19.0019216Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:44:19.0053108Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:44:19.0086849Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:44:19.0119667Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:44:19.0152862Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:44:19.0214693Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:44:19.0227390Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0236742Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T07:44:19.0265257Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:44:19.0475680Z Entering 'android/libs/fbjni' 2024-06-26T07:44:19.0496206Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0519865Z Entering 'third_party/FP16' 2024-06-26T07:44:19.0542287Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0566236Z Entering 'third_party/FXdiv' 2024-06-26T07:44:19.0587166Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0612065Z Entering 'third_party/NNPACK' 2024-06-26T07:44:19.0632308Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0654806Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:44:19.0676582Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0700164Z Entering 'third_party/XNNPACK' 2024-06-26T07:44:19.0722202Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0758379Z Entering 'third_party/benchmark' 2024-06-26T07:44:19.0780321Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0803180Z Entering 'third_party/cpp-httplib' 2024-06-26T07:44:19.0824642Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0848062Z Entering 'third_party/cpuinfo' 2024-06-26T07:44:19.0870153Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0893467Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:44:19.0915427Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0938104Z Entering 'third_party/cutlass' 2024-06-26T07:44:19.0959609Z http.https://github.com/.extraheader 2024-06-26T07:44:19.0986858Z Entering 'third_party/eigen' 2024-06-26T07:44:19.1008870Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1034788Z Entering 'third_party/fbgemm' 2024-06-26T07:44:19.1056256Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1079115Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:44:19.1100571Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1123254Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:44:19.1144651Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1167139Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:44:19.1188249Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1216474Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:44:19.1238035Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1262475Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:44:19.1283229Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1307917Z Entering 'third_party/flatbuffers' 2024-06-26T07:44:19.1328989Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1353346Z Entering 'third_party/fmt' 2024-06-26T07:44:19.1375996Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1399978Z Entering 'third_party/foxi' 2024-06-26T07:44:19.1420829Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1445849Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:44:19.1467853Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1490542Z Entering 'third_party/gloo' 2024-06-26T07:44:19.1511652Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1534672Z Entering 'third_party/googletest' 2024-06-26T07:44:19.1557316Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1579314Z Entering 'third_party/ideep' 2024-06-26T07:44:19.1601150Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1624378Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:44:19.1645720Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1674482Z Entering 'third_party/ittapi' 2024-06-26T07:44:19.1697807Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1721779Z Entering 'third_party/kineto' 2024-06-26T07:44:19.1743326Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1766882Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:44:19.1787837Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1812591Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:44:19.1833484Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1857534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:44:19.1878781Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1901343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:44:19.1922603Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1946283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:44:19.1967242Z http.https://github.com/.extraheader 2024-06-26T07:44:19.1989080Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:44:19.2010965Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2035270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:44:19.2056781Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2080619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:44:19.2101331Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2124882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:44:19.2145571Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2170153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:44:19.2190313Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2213556Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:44:19.2234575Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2257427Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:44:19.2278775Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2303695Z Entering 'third_party/mimalloc' 2024-06-26T07:44:19.2326672Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2350278Z Entering 'third_party/nccl/nccl' 2024-06-26T07:44:19.2372861Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2397394Z Entering 'third_party/nlohmann' 2024-06-26T07:44:19.2419642Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2444097Z Entering 'third_party/onnx' 2024-06-26T07:44:19.2464631Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2499278Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:44:19.2520993Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2544053Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:44:19.2565069Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2588594Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:44:19.2611487Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2636019Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:44:19.2657435Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2679630Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:44:19.2701290Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2725049Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:44:19.2746074Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2769919Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:44:19.2790438Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2814197Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:44:19.2835136Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2858540Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:44:19.2879968Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2903745Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:44:19.2926226Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2948729Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:44:19.2970499Z http.https://github.com/.extraheader 2024-06-26T07:44:19.2996657Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:44:19.3016685Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3040757Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:44:19.3062143Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3101005Z Entering 'third_party/pocketfft' 2024-06-26T07:44:19.3122803Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3145238Z Entering 'third_party/protobuf' 2024-06-26T07:44:19.3167408Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3191647Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:44:19.3213168Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3236251Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:44:19.3257712Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3281636Z Entering 'third_party/psimd' 2024-06-26T07:44:19.3302713Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3324757Z Entering 'third_party/pthreadpool' 2024-06-26T07:44:19.3346017Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3369449Z Entering 'third_party/pybind11' 2024-06-26T07:44:19.3391002Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3415930Z Entering 'third_party/python-peachpy' 2024-06-26T07:44:19.3437484Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3461172Z Entering 'third_party/sleef' 2024-06-26T07:44:19.3482465Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3506173Z Entering 'third_party/tensorpipe' 2024-06-26T07:44:19.3528097Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3550102Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:44:19.3571323Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3594692Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:44:19.3616528Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3640449Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:44:19.3661298Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3684340Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:44:19.3706199Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3728168Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:44:19.3748536Z http.https://github.com/.extraheader 2024-06-26T07:44:19.3854907Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T07:44:19.3872611Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:44:19.3880550Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:44:19.3881078Z ##[endgroup] 2024-06-26T07:44:20.3123410Z Cleaning up orphan processes