2024-06-26T06:59:58.1687413Z Current runner version: '2.317.0' 2024-06-26T06:59:58.1693695Z Runner name: 'i-0b34867d3f3dcf691' 2024-06-26T06:59:58.1694605Z Runner group name: 'Default' 2024-06-26T06:59:58.1695545Z Machine name: 'ip-10-0-56-211' 2024-06-26T06:59:58.1700692Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T06:59:58.1702704Z Actions: read 2024-06-26T06:59:58.1703288Z Attestations: read 2024-06-26T06:59:58.1703869Z Checks: read 2024-06-26T06:59:58.1704463Z Contents: read 2024-06-26T06:59:58.1704993Z Deployments: read 2024-06-26T06:59:58.1705575Z Discussions: read 2024-06-26T06:59:58.1706149Z Issues: read 2024-06-26T06:59:58.1706665Z Metadata: read 2024-06-26T06:59:58.1707255Z Packages: read 2024-06-26T06:59:58.1707816Z Pages: read 2024-06-26T06:59:58.1708359Z PullRequests: read 2024-06-26T06:59:58.1708943Z RepositoryProjects: read 2024-06-26T06:59:58.1709611Z SecurityEvents: read 2024-06-26T06:59:58.1710196Z Statuses: read 2024-06-26T06:59:58.1710869Z ##[endgroup] 2024-06-26T06:59:58.1715827Z Secret source: Actions 2024-06-26T06:59:58.1716673Z Prepare workflow directory 2024-06-26T06:59:58.5245455Z Prepare all required actions 2024-06-26T06:59:58.5430671Z Getting action download info 2024-06-26T06:59:58.7180317Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T06:59:59.0494945Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:00:01.6841015Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:00:01.8451123Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:00:02.1026137Z Getting action download info 2024-06-26T07:00:02.2252475Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:00:02.3978092Z Getting action download info 2024-06-26T07:00:02.4865488Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:00:02.6406582Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:00:02.6408864Z ##[group] Inputs 2024-06-26T07:00:02.6409420Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:00:02.6418238Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:00:02.6427914Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:00:02.6429166Z sync-tag: 2024-06-26T07:00:02.6430057Z timeout-minutes: 240 2024-06-26T07:00:02.6430474Z use-gha: 2024-06-26T07:00:02.6430963Z dashboard-tag: 2024-06-26T07:00:02.6431419Z s3-bucket: gha-artifacts 2024-06-26T07:00:02.6431854Z aws-role-to-assume: 2024-06-26T07:00:02.6432278Z ##[endgroup] 2024-06-26T07:00:02.6433288Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:00:02.7019791Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:00:02.7161192Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:00:02.7171744Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:00:02.7172352Z ##[endgroup] 2024-06-26T07:00:03.3323838Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:00:03.3324492Z with: 2024-06-26T07:00:03.3325196Z github-secret: *** 2024-06-26T07:00:03.3326239Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:00:03.3327319Z activate-with-label: false 2024-06-26T07:00:03.3327760Z label: with-ssh 2024-06-26T07:00:03.3328159Z remove-existing-keys: true 2024-06-26T07:00:03.3328603Z fail-silently: true 2024-06-26T07:00:03.3328996Z env: 2024-06-26T07:00:03.3329365Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:03.3329790Z ##[endgroup] 2024-06-26T07:00:03.4207241Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:00:03.4208359Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:00:03.7309709Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:00:03.8080554Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:00:03.8081526Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:00:03.8818857Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:00:03.8927794Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:00:03.8928358Z with: 2024-06-26T07:00:03.8928644Z submodules: recursive 2024-06-26T07:00:03.8928978Z fetch-depth: 0 2024-06-26T07:00:03.8929275Z env: 2024-06-26T07:00:03.8929555Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:03.8929891Z ##[endgroup] 2024-06-26T07:00:03.9123983Z ##[group]Run retry () { 2024-06-26T07:00:03.9124448Z retry () { 2024-06-26T07:00:03.9125012Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:00:03.9125606Z } 2024-06-26T07:00:03.9125976Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:00:03.9126455Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:00:03.9126985Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:00:03.9127489Z else 2024-06-26T07:00:03.9127888Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:00:03.9128363Z fi 2024-06-26T07:00:03.9128773Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:00:03.9136832Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:00:03.9137387Z env: 2024-06-26T07:00:03.9137723Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:03.9138116Z NO_SUDO: 2024-06-26T07:00:03.9138443Z ##[endgroup] 2024-06-26T07:00:03.9161229Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:00:06.1078107Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:00:06.1078547Z with: 2024-06-26T07:00:06.1078875Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:06.1079314Z fetch-depth: 0 2024-06-26T07:00:06.1079631Z submodules: recursive 2024-06-26T07:00:06.1079974Z quiet-checkout: true 2024-06-26T07:00:06.1080325Z repository: pytorch/pytorch 2024-06-26T07:00:06.1080816Z token: *** 2024-06-26T07:00:06.1081101Z ssh-strict: true 2024-06-26T07:00:06.1081426Z persist-credentials: true 2024-06-26T07:00:06.1081777Z clean: true 2024-06-26T07:00:06.1082100Z sparse-checkout-cone-mode: true 2024-06-26T07:00:06.1082487Z lfs: false 2024-06-26T07:00:06.1082786Z set-safe-directory: true 2024-06-26T07:00:06.1083121Z env: 2024-06-26T07:00:06.1083382Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:06.1083728Z ##[endgroup] 2024-06-26T07:00:06.2001693Z Syncing repository: pytorch/pytorch 2024-06-26T07:00:06.2003244Z ##[group]Getting Git version info 2024-06-26T07:00:06.2004015Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:00:06.2004914Z [command]/usr/bin/git version 2024-06-26T07:00:06.2005287Z git version 2.40.1 2024-06-26T07:00:06.2006658Z ##[endgroup] 2024-06-26T07:00:06.2018179Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/09816a1e-3d6b-45ee-90a7-beb4d9ccfaa0' before making global git config changes 2024-06-26T07:00:06.2019437Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:00:06.2022962Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:00:06.2058504Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:00:06.2063075Z ##[group]Initializing the repository 2024-06-26T07:00:06.2066822Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:00:06.2090828Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:00:06.2091861Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:00:06.2092691Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:00:06.2093249Z hint: 2024-06-26T07:00:06.2093847Z hint: git config --global init.defaultBranch 2024-06-26T07:00:06.2094325Z hint: 2024-06-26T07:00:06.2094839Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:00:06.2095674Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:00:06.2096473Z hint: 2024-06-26T07:00:06.2096798Z hint: git branch -m 2024-06-26T07:00:06.2097525Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:00:06.2099564Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:00:06.2125778Z ##[endgroup] 2024-06-26T07:00:06.2126384Z ##[group]Disabling automatic garbage collection 2024-06-26T07:00:06.2128778Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:00:06.2154252Z ##[endgroup] 2024-06-26T07:00:06.2155273Z ##[group]Setting up auth 2024-06-26T07:00:06.2161044Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:00:06.2186663Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:00:06.2385958Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:00:06.2411888Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:00:06.2610217Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:00:06.2653993Z ##[endgroup] 2024-06-26T07:00:06.2654836Z ##[group]Fetching the repository 2024-06-26T07:00:06.2661391Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:00:08.4936981Z remote: Enumerating objects: 987392 2024-06-26T07:00:08.4937549Z remote: Enumerating objects: 990209, done. 2024-06-26T07:00:08.4938668Z remote: Counting objects: 0% (1/2817) 2024-06-26T07:00:08.4941666Z remote: Counting objects: 1% (29/2817) 2024-06-26T07:00:08.4942411Z remote: Counting objects: 2% (57/2817) 2024-06-26T07:00:08.4943018Z remote: Counting objects: 3% (85/2817) 2024-06-26T07:00:08.4943641Z remote: Counting objects: 4% (113/2817) 2024-06-26T07:00:08.4944246Z remote: Counting objects: 5% (141/2817) 2024-06-26T07:00:08.4944750Z remote: Counting objects: 6% (170/2817) 2024-06-26T07:00:08.4945320Z remote: Counting objects: 7% (198/2817) 2024-06-26T07:00:08.4946003Z remote: Counting objects: 8% (226/2817) 2024-06-26T07:00:08.4946708Z remote: Counting objects: 9% (254/2817) 2024-06-26T07:00:08.4947229Z remote: Counting objects: 10% (282/2817) 2024-06-26T07:00:08.4947780Z remote: Counting objects: 11% (310/2817) 2024-06-26T07:00:08.4948432Z remote: Counting objects: 12% (339/2817) 2024-06-26T07:00:08.4948941Z remote: Counting objects: 13% (367/2817) 2024-06-26T07:00:08.4949439Z remote: Counting objects: 14% (395/2817) 2024-06-26T07:00:08.4950004Z remote: Counting objects: 15% (423/2817) 2024-06-26T07:00:08.4950627Z remote: Counting objects: 16% (451/2817) 2024-06-26T07:00:08.4951295Z remote: Counting objects: 17% (479/2817) 2024-06-26T07:00:08.4951806Z remote: Counting objects: 18% (508/2817) 2024-06-26T07:00:08.4952337Z remote: Counting objects: 19% (536/2817) 2024-06-26T07:00:08.4953000Z remote: Counting objects: 20% (564/2817) 2024-06-26T07:00:08.4953640Z remote: Counting objects: 21% (592/2817) 2024-06-26T07:00:08.4954246Z remote: Counting objects: 22% (620/2817) 2024-06-26T07:00:08.4955107Z remote: Counting objects: 23% (648/2817) 2024-06-26T07:00:08.4955746Z remote: Counting objects: 24% (677/2817) 2024-06-26T07:00:08.4956322Z remote: Counting objects: 25% (705/2817) 2024-06-26T07:00:08.4956881Z remote: Counting objects: 26% (733/2817) 2024-06-26T07:00:08.4957540Z remote: Counting objects: 27% (761/2817) 2024-06-26T07:00:08.4958264Z remote: Counting objects: 28% (789/2817) 2024-06-26T07:00:08.4958775Z remote: Counting objects: 29% (817/2817) 2024-06-26T07:00:08.4959453Z remote: Counting objects: 30% (846/2817) 2024-06-26T07:00:08.4960042Z remote: Counting objects: 31% (874/2817) 2024-06-26T07:00:08.4960548Z remote: Counting objects: 32% (902/2817) 2024-06-26T07:00:08.4961071Z remote: Counting objects: 33% (930/2817) 2024-06-26T07:00:08.4961677Z remote: Counting objects: 34% (958/2817) 2024-06-26T07:00:08.4962353Z remote: Counting objects: 35% (986/2817) 2024-06-26T07:00:08.4962887Z remote: Counting objects: 36% (1015/2817) 2024-06-26T07:00:08.4963419Z remote: Counting objects: 37% (1043/2817) 2024-06-26T07:00:08.4964043Z remote: Counting objects: 38% (1071/2817) 2024-06-26T07:00:08.4964691Z remote: Counting objects: 39% (1099/2817) 2024-06-26T07:00:08.4965217Z remote: Counting objects: 40% (1127/2817) 2024-06-26T07:00:08.4965737Z remote: Counting objects: 41% (1155/2817) 2024-06-26T07:00:08.4966254Z remote: Counting objects: 42% (1184/2817) 2024-06-26T07:00:08.4966934Z remote: Counting objects: 43% (1212/2817) 2024-06-26T07:00:08.4967545Z remote: Counting objects: 44% (1240/2817) 2024-06-26T07:00:08.4968065Z remote: Counting objects: 45% (1268/2817) 2024-06-26T07:00:08.4968727Z remote: Counting objects: 46% (1296/2817) 2024-06-26T07:00:08.4969355Z remote: Counting objects: 47% (1324/2817) 2024-06-26T07:00:08.4970040Z remote: Counting objects: 48% (1353/2817) 2024-06-26T07:00:08.4970565Z remote: Counting objects: 49% (1381/2817) 2024-06-26T07:00:08.4971087Z remote: Counting objects: 50% (1409/2817) 2024-06-26T07:00:08.4971621Z remote: Counting objects: 51% (1437/2817) 2024-06-26T07:00:08.4972208Z remote: Counting objects: 52% (1465/2817) 2024-06-26T07:00:08.4972787Z remote: Counting objects: 53% (1494/2817) 2024-06-26T07:00:08.4973309Z remote: Counting objects: 54% (1522/2817) 2024-06-26T07:00:08.4973833Z remote: Counting objects: 55% (1550/2817) 2024-06-26T07:00:08.4974421Z remote: Counting objects: 56% (1578/2817) 2024-06-26T07:00:08.4975106Z remote: Counting objects: 57% (1606/2817) 2024-06-26T07:00:08.4975639Z remote: Counting objects: 58% (1634/2817) 2024-06-26T07:00:08.4976157Z remote: Counting objects: 59% (1663/2817) 2024-06-26T07:00:08.4976671Z remote: Counting objects: 60% (1691/2817) 2024-06-26T07:00:08.4977321Z remote: Counting objects: 61% (1719/2817) 2024-06-26T07:00:08.4977961Z remote: Counting objects: 62% (1747/2817) 2024-06-26T07:00:08.4978492Z remote: Counting objects: 63% (1775/2817) 2024-06-26T07:00:08.4979010Z remote: Counting objects: 64% (1803/2817) 2024-06-26T07:00:08.4979513Z remote: Counting objects: 65% (1832/2817) 2024-06-26T07:00:08.4980033Z remote: Counting objects: 66% (1860/2817) 2024-06-26T07:00:08.4980559Z remote: Counting objects: 67% (1888/2817) 2024-06-26T07:00:08.4981239Z remote: Counting objects: 68% (1916/2817) 2024-06-26T07:00:08.4981921Z remote: Counting objects: 69% (1944/2817) 2024-06-26T07:00:08.4982450Z remote: Counting objects: 70% (1972/2817) 2024-06-26T07:00:08.4982969Z remote: Counting objects: 71% (2001/2817) 2024-06-26T07:00:08.4983480Z remote: Counting objects: 72% (2029/2817) 2024-06-26T07:00:08.4983988Z remote: Counting objects: 73% (2057/2817) 2024-06-26T07:00:08.4984505Z remote: Counting objects: 74% (2085/2817) 2024-06-26T07:00:08.4985029Z remote: Counting objects: 75% (2113/2817) 2024-06-26T07:00:08.4985542Z remote: Counting objects: 76% (2141/2817) 2024-06-26T07:00:08.4986052Z remote: Counting objects: 77% (2170/2817) 2024-06-26T07:00:08.4986569Z remote: Counting objects: 78% (2198/2817) 2024-06-26T07:00:08.4987190Z remote: Counting objects: 79% (2226/2817) 2024-06-26T07:00:08.4988144Z remote: Counting objects: 80% (2254/2817) 2024-06-26T07:00:08.4988869Z remote: Counting objects: 81% (2282/2817) 2024-06-26T07:00:08.4989530Z remote: Counting objects: 82% (2310/2817) 2024-06-26T07:00:08.4990155Z remote: Counting objects: 83% (2339/2817) 2024-06-26T07:00:08.4990868Z remote: Counting objects: 84% (2367/2817) 2024-06-26T07:00:08.4991379Z remote: Counting objects: 85% (2395/2817) 2024-06-26T07:00:08.5003087Z remote: Counting objects: 86% (2423/2817) 2024-06-26T07:00:08.5003799Z remote: Counting objects: 87% (2451/2817) 2024-06-26T07:00:08.5004372Z remote: Counting objects: 88% (2479/2817) 2024-06-26T07:00:08.5004883Z remote: Counting objects: 89% (2508/2817) 2024-06-26T07:00:08.5005392Z remote: Counting objects: 90% (2536/2817) 2024-06-26T07:00:08.5005906Z remote: Counting objects: 91% (2564/2817) 2024-06-26T07:00:08.5006428Z remote: Counting objects: 92% (2592/2817) 2024-06-26T07:00:08.5006942Z remote: Counting objects: 93% (2620/2817) 2024-06-26T07:00:08.5007445Z remote: Counting objects: 94% (2648/2817) 2024-06-26T07:00:08.5008370Z remote: Counting objects: 95% (2677/2817) 2024-06-26T07:00:08.5009080Z remote: Counting objects: 96% (2705/2817) 2024-06-26T07:00:08.5009776Z remote: Counting objects: 97% (2733/2817) 2024-06-26T07:00:08.5010288Z remote: Counting objects: 98% (2761/2817) 2024-06-26T07:00:08.5010804Z remote: Counting objects: 99% (2789/2817) 2024-06-26T07:00:08.5011310Z remote: Counting objects: 100% (2817/2817) 2024-06-26T07:00:08.5011939Z remote: Counting objects: 100% (2817/2817), done. 2024-06-26T07:00:08.5170519Z remote: Compressing objects: 0% (1/1140) 2024-06-26T07:00:08.5749278Z remote: Compressing objects: 1% (12/1140) 2024-06-26T07:00:08.6177590Z remote: Compressing objects: 2% (23/1140) 2024-06-26T07:00:08.6500843Z remote: Compressing objects: 3% (35/1140) 2024-06-26T07:00:08.6688787Z remote: Compressing objects: 4% (46/1140) 2024-06-26T07:00:08.7054759Z remote: Compressing objects: 5% (57/1140) 2024-06-26T07:00:08.7376267Z remote: Compressing objects: 6% (69/1140) 2024-06-26T07:00:08.8410472Z remote: Compressing objects: 7% (80/1140) 2024-06-26T07:00:08.9075758Z remote: Compressing objects: 8% (92/1140) 2024-06-26T07:00:08.9547266Z remote: Compressing objects: 9% (103/1140) 2024-06-26T07:00:08.9723491Z remote: Compressing objects: 10% (114/1140) 2024-06-26T07:00:08.9819948Z remote: Compressing objects: 11% (126/1140) 2024-06-26T07:00:08.9892996Z remote: Compressing objects: 12% (137/1140) 2024-06-26T07:00:08.9935430Z remote: Compressing objects: 13% (149/1140) 2024-06-26T07:00:08.9936147Z remote: Compressing objects: 14% (160/1140) 2024-06-26T07:00:09.0003048Z remote: Compressing objects: 15% (171/1140) 2024-06-26T07:00:09.0003772Z remote: Compressing objects: 16% (183/1140) 2024-06-26T07:00:09.0005207Z remote: Compressing objects: 17% (194/1140) 2024-06-26T07:00:09.0028543Z remote: Compressing objects: 18% (206/1140) 2024-06-26T07:00:09.0051039Z remote: Compressing objects: 19% (217/1140) 2024-06-26T07:00:09.0073908Z remote: Compressing objects: 20% (228/1140) 2024-06-26T07:00:09.0092981Z remote: Compressing objects: 21% (240/1140) 2024-06-26T07:00:09.0114892Z remote: Compressing objects: 22% (251/1140) 2024-06-26T07:00:09.0124439Z remote: Compressing objects: 23% (263/1140) 2024-06-26T07:00:09.0132079Z remote: Compressing objects: 24% (274/1140) 2024-06-26T07:00:09.0141431Z remote: Compressing objects: 25% (285/1140) 2024-06-26T07:00:09.0151425Z remote: Compressing objects: 26% (297/1140) 2024-06-26T07:00:09.0154136Z remote: Compressing objects: 27% (308/1140) 2024-06-26T07:00:09.0162347Z remote: Compressing objects: 28% (320/1140) 2024-06-26T07:00:09.0172069Z remote: Compressing objects: 29% (331/1140) 2024-06-26T07:00:09.0179169Z remote: Compressing objects: 30% (342/1140) 2024-06-26T07:00:09.0181768Z remote: Compressing objects: 31% (354/1140) 2024-06-26T07:00:09.0199638Z remote: Compressing objects: 32% (365/1140) 2024-06-26T07:00:09.0212199Z remote: Compressing objects: 33% (377/1140) 2024-06-26T07:00:09.0228850Z remote: Compressing objects: 34% (388/1140) 2024-06-26T07:00:09.0240437Z remote: Compressing objects: 35% (399/1140) 2024-06-26T07:00:09.0266961Z remote: Compressing objects: 36% (411/1140) 2024-06-26T07:00:09.0285657Z remote: Compressing objects: 37% (422/1140) 2024-06-26T07:00:09.0301103Z remote: Compressing objects: 38% (434/1140) 2024-06-26T07:00:09.0313670Z remote: Compressing objects: 39% (445/1140) 2024-06-26T07:00:09.0331026Z remote: Compressing objects: 40% (456/1140) 2024-06-26T07:00:09.0355127Z remote: Compressing objects: 41% (468/1140) 2024-06-26T07:00:09.0377945Z remote: Compressing objects: 42% (479/1140) 2024-06-26T07:00:09.0387615Z remote: Compressing objects: 43% (491/1140) 2024-06-26T07:00:09.0395674Z remote: Compressing objects: 44% (502/1140) 2024-06-26T07:00:09.0399153Z remote: Compressing objects: 45% (513/1140) 2024-06-26T07:00:09.0410154Z remote: Compressing objects: 46% (525/1140) 2024-06-26T07:00:09.0432407Z remote: Compressing objects: 47% (536/1140) 2024-06-26T07:00:09.0435433Z remote: Compressing objects: 48% (548/1140) 2024-06-26T07:00:09.0440212Z remote: Compressing objects: 49% (559/1140) 2024-06-26T07:00:09.0448796Z remote: Compressing objects: 50% (570/1140) 2024-06-26T07:00:09.0461829Z remote: Compressing objects: 51% (582/1140) 2024-06-26T07:00:09.0490703Z remote: Compressing objects: 52% (593/1140) 2024-06-26T07:00:09.0505244Z remote: Compressing objects: 53% (605/1140) 2024-06-26T07:00:09.0536265Z remote: Compressing objects: 54% (616/1140) 2024-06-26T07:00:09.0543469Z remote: Compressing objects: 55% (627/1140) 2024-06-26T07:00:09.0549552Z remote: Compressing objects: 56% (639/1140) 2024-06-26T07:00:09.0560200Z remote: Compressing objects: 57% (650/1140) 2024-06-26T07:00:09.0579147Z remote: Compressing objects: 58% (662/1140) 2024-06-26T07:00:09.0583855Z remote: Compressing objects: 59% (673/1140) 2024-06-26T07:00:09.0592691Z remote: Compressing objects: 60% (684/1140) 2024-06-26T07:00:09.0600723Z remote: Compressing objects: 61% (696/1140) 2024-06-26T07:00:09.0615213Z remote: Compressing objects: 62% (707/1140) 2024-06-26T07:00:09.0627352Z remote: Compressing objects: 63% (719/1140) 2024-06-26T07:00:09.0636892Z remote: Compressing objects: 64% (730/1140) 2024-06-26T07:00:09.0646126Z remote: Compressing objects: 65% (741/1140) 2024-06-26T07:00:09.0654276Z remote: Compressing objects: 66% (753/1140) 2024-06-26T07:00:09.0661753Z remote: Compressing objects: 67% (764/1140) 2024-06-26T07:00:09.0665171Z remote: Compressing objects: 68% (776/1140) 2024-06-26T07:00:09.0674610Z remote: Compressing objects: 69% (787/1140) 2024-06-26T07:00:09.0681617Z remote: Compressing objects: 70% (798/1140) 2024-06-26T07:00:09.0684046Z remote: Compressing objects: 71% (810/1140) 2024-06-26T07:00:09.0686468Z remote: Compressing objects: 72% (821/1140) 2024-06-26T07:00:09.0689213Z remote: Compressing objects: 73% (833/1140) 2024-06-26T07:00:09.0689762Z remote: Compressing objects: 74% (844/1140) 2024-06-26T07:00:09.0691865Z remote: Compressing objects: 75% (855/1140) 2024-06-26T07:00:09.0694314Z remote: Compressing objects: 76% (867/1140) 2024-06-26T07:00:09.0695005Z remote: Compressing objects: 77% (878/1140) 2024-06-26T07:00:09.0695560Z remote: Compressing objects: 78% (890/1140) 2024-06-26T07:00:09.0696169Z remote: Compressing objects: 79% (901/1140) 2024-06-26T07:00:09.0697229Z remote: Compressing objects: 80% (912/1140) 2024-06-26T07:00:09.0699580Z remote: Compressing objects: 81% (924/1140) 2024-06-26T07:00:09.0702043Z remote: Compressing objects: 82% (935/1140) 2024-06-26T07:00:09.0702587Z remote: Compressing objects: 83% (947/1140) 2024-06-26T07:00:09.0704706Z remote: Compressing objects: 84% (958/1140) 2024-06-26T07:00:09.0705257Z remote: Compressing objects: 85% (969/1140) 2024-06-26T07:00:09.0707323Z remote: Compressing objects: 86% (981/1140) 2024-06-26T07:00:09.0709823Z remote: Compressing objects: 87% (992/1140) 2024-06-26T07:00:09.0710384Z remote: Compressing objects: 88% (1004/1140) 2024-06-26T07:00:09.0712599Z remote: Compressing objects: 89% (1015/1140) 2024-06-26T07:00:09.0715222Z remote: Compressing objects: 90% (1026/1140) 2024-06-26T07:00:09.0715777Z remote: Compressing objects: 91% (1038/1140) 2024-06-26T07:00:09.0716321Z remote: Compressing objects: 92% (1049/1140) 2024-06-26T07:00:09.0717420Z remote: Compressing objects: 93% (1061/1140) 2024-06-26T07:00:09.0720314Z remote: Compressing objects: 94% (1072/1140) 2024-06-26T07:00:09.0721005Z remote: Compressing objects: 95% (1083/1140) 2024-06-26T07:00:09.0721553Z remote: Compressing objects: 96% (1095/1140) 2024-06-26T07:00:09.0722586Z remote: Compressing objects: 97% (1106/1140) 2024-06-26T07:00:09.0725380Z remote: Compressing objects: 98% (1118/1140) 2024-06-26T07:00:09.0725943Z remote: Compressing objects: 99% (1129/1140) 2024-06-26T07:00:09.0726483Z remote: Compressing objects: 100% (1140/1140) 2024-06-26T07:00:09.0727065Z remote: Compressing objects: 100% (1140/1140), done. 2024-06-26T07:00:27.8066427Z remote: Total 990209 (delta 2280), reused 2016 (delta 1676), pack-reused 987392 2024-06-26T07:00:53.9661553Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:00:53.9681042Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:53.9684623Z ##[endgroup] 2024-06-26T07:00:53.9686557Z ##[group]Determining the checkout info 2024-06-26T07:00:53.9688513Z ##[endgroup] 2024-06-26T07:00:53.9689243Z ##[group]Checking out the ref 2024-06-26T07:00:53.9690227Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:55.1904296Z ##[endgroup] 2024-06-26T07:00:55.1905498Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:00:55.1907387Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:00:55.1947313Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:00:55.1973741Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:00:55.1999487Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:00:55.2017048Z ##[endgroup] 2024-06-26T07:00:55.2017857Z ##[group]Fetching submodules 2024-06-26T07:00:55.2021778Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:00:55.2236970Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:00:55.2455713Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:00:55.2457192Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:00:55.2458596Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:00:55.2460133Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:00:55.2461842Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:00:55.2463805Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:00:55.2465577Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:00:55.2467953Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:00:55.2470014Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:00:55.2472721Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:00:55.2475305Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:00:55.2477927Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:00:55.2480305Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:00:55.2482937Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:00:55.2485332Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:00:55.2488047Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:00:55.2490960Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:00:55.2493609Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:00:55.2496528Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:00:55.2499335Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:00:55.2502466Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:00:55.2505530Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:00:55.2508851Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:00:55.2512186Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:00:55.2515665Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:00:55.2519135Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:00:55.2522819Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:00:55.2526066Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:00:55.2529687Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:00:55.2533235Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:00:55.2537196Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:00:55.2540765Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:00:55.2544497Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:00:55.2548321Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:00:55.2552520Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:00:55.2573642Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:00:55.4835191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:00:55.6609097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:00:55.8200915Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:00:56.0256537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:00:57.9895392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:01:08.3167774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:01:08.6652295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:01:09.1622875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:01:09.7356133Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:01:10.8891174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:01:12.5858186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:01:18.4825869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:01:19.7433174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:01:21.5351074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:01:22.7294784Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:01:22.8771925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:01:23.2531940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:01:23.5759155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:01:24.5328549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:01:24.8682676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:01:25.1627538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:01:26.6321300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:01:27.4279853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:01:28.1026115Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:01:33.7173265Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:01:36.1654089Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:01:41.8573352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:01:42.0958712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:01:50.6934299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:01:50.8509891Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:01:51.0930964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:01:52.1319743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:01:52.4133459Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:01:52.9747475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:01:53.4319704Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:01:53.4403539Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:01:53.4465038Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:01:53.4646247Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:01:53.4948024Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:01:54.2711603Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:01:54.2890154Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:01:54.3233263Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:01:54.4085463Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:01:54.4353201Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:01:54.8628910Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:01:55.0755024Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:01:55.1404183Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:01:55.1417469Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:01:55.1420457Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:01:55.1421945Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:01:55.1423466Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:01:55.1425126Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:01:55.1442892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:01:56.0649599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:01:56.6539376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:01:58.3424722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:01:59.3657124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:01:59.6835065Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:01:59.7681135Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:02:00.1100500Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:02:00.1653464Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:02:00.1737181Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:02:00.2651888Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:02:00.2930678Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:02:00.2992199Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:02:00.3295439Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:02:00.3492487Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:02:00.3855013Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:02:00.3955449Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:02:00.3967014Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:00.3983871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:02:12.0944578Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:02:12.1080750Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:02:12.1863309Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:02:12.1875286Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:12.1877309Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:12.1879299Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:12.1897109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:02:12.7005450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:02:13.9523762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:02:14.9433620Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:02:14.9443401Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:14.9445343Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:14.9447238Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:14.9449118Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:14.9450990Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:14.9453161Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:14.9455314Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:14.9457258Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:14.9475009Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:02:15.9756505Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:02:16.3440046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:02:17.5913880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:02:17.9003099Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:02:18.4182288Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:02:19.5533285Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:02:25.1110770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:02:25.4364050Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:02:25.4498836Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:02:25.4789854Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:02:25.4884412Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:02:25.4893449Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:25.4908414Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:02:25.8265445Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:02:25.8398875Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:02:25.8732077Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:02:25.9536006Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:02:25.9650175Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:02:25.9943112Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:02:26.0428040Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:02:26.0737023Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:02:26.0937818Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:02:26.1789615Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:02:26.4510163Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:02:26.4540035Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:26.4543245Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:26.4556659Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:02:26.9125266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:02:28.0675915Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:02:28.0955726Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:02:28.1503051Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:02:28.1515246Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:28.1521669Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:28.1523254Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:28.1524839Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:28.1526703Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:28.1528699Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:28.1531063Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:28.1532668Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:28.1546396Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:02:28.5859789Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:02:29.5688009Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:02:29.9095211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:02:36.0584224Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:02:36.3197320Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:02:36.5356761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:02:37.3110878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:02:44.6562192Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:02:44.6895093Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:02:44.7019666Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:02:44.7862895Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:02:44.7956142Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:02:44.8061891Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:02:44.8173490Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:02:44.8184148Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:44.8186139Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:44.8202853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:02:46.5691768Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:02:47.7439378Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:02:47.7803282Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:02:48.1488609Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:02:48.1573546Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:02:48.3761460Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:02:48.3773923Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:48.3776075Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:48.3794975Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:02:48.7290955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:02:49.7424757Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:02:49.8055432Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:02:49.8116009Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:02:49.8203536Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:02:49.8490203Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:02:49.8710885Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:02:49.9044504Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:02:49.9248078Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:02:49.9260168Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:49.9262569Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:49.9264839Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:49.9267255Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:49.9284845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:02:50.8837815Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:02:51.3571199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:02:52.3506931Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:02:53.3711274Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:02:53.3823892Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:02:53.4387158Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:02:53.4617769Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:02:53.4628854Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:53.4646425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:02:53.7048028Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:02:53.7071153Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:02:53.7270075Z Entering 'android/libs/fbjni' 2024-06-26T07:02:53.7295578Z Entering 'third_party/FP16' 2024-06-26T07:02:53.7321247Z Entering 'third_party/FXdiv' 2024-06-26T07:02:53.7346412Z Entering 'third_party/NNPACK' 2024-06-26T07:02:53.7376138Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:53.7402397Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:53.7438760Z Entering 'third_party/benchmark' 2024-06-26T07:02:53.7468350Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:53.7493768Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:53.7520414Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:53.7548700Z Entering 'third_party/cutlass' 2024-06-26T07:02:53.7581718Z Entering 'third_party/eigen' 2024-06-26T07:02:53.7611067Z Entering 'third_party/fbgemm' 2024-06-26T07:02:53.7638810Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:53.7666097Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:53.7693576Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:53.7723990Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:53.7751487Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:53.7778247Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:53.7807362Z Entering 'third_party/fmt' 2024-06-26T07:02:53.7836882Z Entering 'third_party/foxi' 2024-06-26T07:02:53.7867242Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:53.7894497Z Entering 'third_party/gloo' 2024-06-26T07:02:53.7923328Z Entering 'third_party/googletest' 2024-06-26T07:02:53.7950174Z Entering 'third_party/ideep' 2024-06-26T07:02:53.7977568Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:53.8009851Z Entering 'third_party/ittapi' 2024-06-26T07:02:53.8039194Z Entering 'third_party/kineto' 2024-06-26T07:02:53.8067590Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:53.8096145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:53.8123576Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:53.8149443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:53.8178622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:53.8204010Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:53.8230559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:53.8259759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:53.8285332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:53.8315216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:53.8345546Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:53.8372500Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:53.8403701Z Entering 'third_party/mimalloc' 2024-06-26T07:02:53.8431068Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:53.8458714Z Entering 'third_party/nlohmann' 2024-06-26T07:02:53.8485974Z Entering 'third_party/onnx' 2024-06-26T07:02:53.8524831Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:53.8555440Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:53.8585376Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:53.8613275Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:53.8639298Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:53.8666447Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:53.8692616Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:53.8720923Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:53.8749130Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:53.8775680Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:53.8801915Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:53.8831640Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:53.8859888Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:53.8903078Z Entering 'third_party/pocketfft' 2024-06-26T07:02:53.8931839Z Entering 'third_party/protobuf' 2024-06-26T07:02:53.8962160Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:53.8988966Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:53.9021173Z Entering 'third_party/psimd' 2024-06-26T07:02:53.9050091Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:53.9076487Z Entering 'third_party/pybind11' 2024-06-26T07:02:53.9103511Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:53.9133148Z Entering 'third_party/sleef' 2024-06-26T07:02:53.9162124Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:53.9189788Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:53.9218358Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:53.9245075Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:53.9273601Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:53.9301241Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:53.9341028Z ##[endgroup] 2024-06-26T07:02:53.9342171Z ##[group]Persisting credentials for submodules 2024-06-26T07:02:53.9347496Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:02:53.9552875Z Entering 'android/libs/fbjni' 2024-06-26T07:02:53.9588238Z Entering 'third_party/FP16' 2024-06-26T07:02:53.9624796Z Entering 'third_party/FXdiv' 2024-06-26T07:02:53.9660988Z Entering 'third_party/NNPACK' 2024-06-26T07:02:53.9695019Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:53.9731548Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:53.9777985Z Entering 'third_party/benchmark' 2024-06-26T07:02:53.9812663Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:53.9843961Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:53.9881193Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:53.9914722Z Entering 'third_party/cutlass' 2024-06-26T07:02:53.9955594Z Entering 'third_party/eigen' 2024-06-26T07:02:53.9988864Z Entering 'third_party/fbgemm' 2024-06-26T07:02:54.0025343Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:54.0062277Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:54.0097754Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:54.0135887Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:54.0171377Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:54.0206284Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:54.0243056Z Entering 'third_party/fmt' 2024-06-26T07:02:54.0278751Z Entering 'third_party/foxi' 2024-06-26T07:02:54.0315958Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:54.0350269Z Entering 'third_party/gloo' 2024-06-26T07:02:54.0387198Z Entering 'third_party/googletest' 2024-06-26T07:02:54.0423774Z Entering 'third_party/ideep' 2024-06-26T07:02:54.0458885Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:54.0498423Z Entering 'third_party/ittapi' 2024-06-26T07:02:54.0534686Z Entering 'third_party/kineto' 2024-06-26T07:02:54.0570058Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:54.0606103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:54.0641508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:54.0676194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:54.0711791Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:54.0746274Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:54.0782244Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:54.0814022Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:54.0846785Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:54.0883776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:54.0917761Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:54.0952673Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:54.0990275Z Entering 'third_party/mimalloc' 2024-06-26T07:02:54.1025928Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:54.1062122Z Entering 'third_party/nlohmann' 2024-06-26T07:02:54.1099693Z Entering 'third_party/onnx' 2024-06-26T07:02:54.1144792Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:54.1179890Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:54.1216383Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:54.1252747Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:54.1286958Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:54.1321551Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:54.1354885Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:54.1391096Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:54.1424321Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:54.1459978Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:54.1493498Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:54.1530064Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:54.1565753Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:54.1615809Z Entering 'third_party/pocketfft' 2024-06-26T07:02:54.1651354Z Entering 'third_party/protobuf' 2024-06-26T07:02:54.1686016Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:54.1720492Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:54.1754089Z Entering 'third_party/psimd' 2024-06-26T07:02:54.1790788Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:54.1826228Z Entering 'third_party/pybind11' 2024-06-26T07:02:54.1863162Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:54.1896598Z Entering 'third_party/sleef' 2024-06-26T07:02:54.1934127Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:54.1967827Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:54.2002899Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:54.2033995Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:54.2069821Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:54.2103252Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:54.2152036Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:02:54.2360431Z Entering 'android/libs/fbjni' 2024-06-26T07:02:54.2390578Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:02:54.2404895Z Entering 'third_party/FP16' 2024-06-26T07:02:54.2438324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:02:54.2452744Z Entering 'third_party/FXdiv' 2024-06-26T07:02:54.2484360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:02:54.2498670Z Entering 'third_party/NNPACK' 2024-06-26T07:02:54.2530357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:02:54.2544753Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:54.2577370Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:02:54.2588939Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:54.2621165Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:02:54.2645820Z Entering 'third_party/benchmark' 2024-06-26T07:02:54.2678661Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:54.2691910Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:54.2724050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:02:54.2737655Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:54.2770776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:02:54.2783360Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:54.2814032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:02:54.2827601Z Entering 'third_party/cutlass' 2024-06-26T07:02:54.2858338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:02:54.2877588Z Entering 'third_party/eigen' 2024-06-26T07:02:54.2911059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:02:54.2927139Z Entering 'third_party/fbgemm' 2024-06-26T07:02:54.2959476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:02:54.2973143Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:54.3005280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:02:54.3018512Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:54.3049468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:02:54.3062339Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:54.3094300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:02:54.3111964Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:54.3143650Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:54.3157153Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:54.3189602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:02:54.3203551Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:54.3236256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:02:54.3251139Z Entering 'third_party/fmt' 2024-06-26T07:02:54.3282415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:02:54.3293255Z Entering 'third_party/foxi' 2024-06-26T07:02:54.3327004Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:02:54.3340162Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:54.3370867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:02:54.3382691Z Entering 'third_party/gloo' 2024-06-26T07:02:54.3415662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:02:54.3428760Z Entering 'third_party/googletest' 2024-06-26T07:02:54.3459986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:54.3472823Z Entering 'third_party/ideep' 2024-06-26T07:02:54.3505344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:02:54.3517905Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:54.3550557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:02:54.3568387Z Entering 'third_party/ittapi' 2024-06-26T07:02:54.3600611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:02:54.3613583Z Entering 'third_party/kineto' 2024-06-26T07:02:54.3646545Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:02:54.3660043Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:54.3692464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:02:54.3705036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:54.3735989Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:02:54.3750353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:54.3785322Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:02:54.3798557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:54.3830728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:02:54.3844055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:54.3875728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:02:54.3888714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:54.3920871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:02:54.3935001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:54.3967613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:02:54.3980365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:54.4012541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:54.4025653Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:54.4059343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:02:54.4072275Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:54.4102986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:02:54.4116939Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:54.4148369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:02:54.4160444Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:54.4191257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:02:54.4204060Z Entering 'third_party/mimalloc' 2024-06-26T07:02:54.4235965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:02:54.4250079Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:54.4281458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:02:54.4295288Z Entering 'third_party/nlohmann' 2024-06-26T07:02:54.4326393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:02:54.4341119Z Entering 'third_party/onnx' 2024-06-26T07:02:54.4374250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:02:54.4398832Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:54.4431448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:54.4445268Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:54.4476790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:54.4490371Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:54.4523769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:02:54.4538431Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:54.4570625Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:54.4582752Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:54.4616006Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:54.4628951Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:54.4662609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:02:54.4674305Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:54.4705710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:02:54.4718621Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:54.4750053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:02:54.4762891Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:54.4798821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:02:54.4811408Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:54.4843200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:02:54.4856073Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:54.4887752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:02:54.4902753Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:54.4936235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:02:54.4950609Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:54.4984636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:02:54.5012129Z Entering 'third_party/pocketfft' 2024-06-26T07:02:54.5045958Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:02:54.5059324Z Entering 'third_party/protobuf' 2024-06-26T07:02:54.5090058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:02:54.5106405Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:54.5138080Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:54.5151041Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:54.5186397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:54.5202312Z Entering 'third_party/psimd' 2024-06-26T07:02:54.5233732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:02:54.5246269Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:54.5278785Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:02:54.5292464Z Entering 'third_party/pybind11' 2024-06-26T07:02:54.5322378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:54.5335192Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:54.5366803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:02:54.5380722Z Entering 'third_party/sleef' 2024-06-26T07:02:54.5410707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:02:54.5421513Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:54.5452763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:02:54.5466723Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:54.5499117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:54.5510408Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:54.5542756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:02:54.5553604Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:54.5584636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:02:54.5596337Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:54.5627438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:54.5639386Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:54.5670651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:02:54.6499561Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:02:54.6706330Z Entering 'android/libs/fbjni' 2024-06-26T07:02:54.6733897Z Entering 'third_party/FP16' 2024-06-26T07:02:54.6763193Z Entering 'third_party/FXdiv' 2024-06-26T07:02:54.6789338Z Entering 'third_party/NNPACK' 2024-06-26T07:02:54.6818928Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:54.6847205Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:54.6886645Z Entering 'third_party/benchmark' 2024-06-26T07:02:54.6917389Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:54.6943776Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:54.6973782Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:54.7002723Z Entering 'third_party/cutlass' 2024-06-26T07:02:54.7036122Z Entering 'third_party/eigen' 2024-06-26T07:02:54.7065813Z Entering 'third_party/fbgemm' 2024-06-26T07:02:54.7093606Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:54.7120990Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:54.7149196Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:54.7182863Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:54.7209922Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:54.7237443Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:54.7265873Z Entering 'third_party/fmt' 2024-06-26T07:02:54.7294840Z Entering 'third_party/foxi' 2024-06-26T07:02:54.7324938Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:54.7352469Z Entering 'third_party/gloo' 2024-06-26T07:02:54.7383075Z Entering 'third_party/googletest' 2024-06-26T07:02:54.7411722Z Entering 'third_party/ideep' 2024-06-26T07:02:54.7439384Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:54.7472684Z Entering 'third_party/ittapi' 2024-06-26T07:02:54.7502669Z Entering 'third_party/kineto' 2024-06-26T07:02:54.7530899Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:54.7559588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:54.7587222Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:54.7614188Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:54.7643172Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:54.7670118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:54.7698867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:54.7727911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:54.7756814Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:54.7786037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:54.7816743Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:54.7846965Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:54.7874001Z Entering 'third_party/mimalloc' 2024-06-26T07:02:54.7904465Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:54.7931725Z Entering 'third_party/nlohmann' 2024-06-26T07:02:54.7963117Z Entering 'third_party/onnx' 2024-06-26T07:02:54.8002210Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:54.8030165Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:54.8063076Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:54.8091993Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:54.8120108Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:54.8149715Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:54.8177835Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:54.8204519Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:54.8228901Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:54.8255355Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:54.8281234Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:54.8309512Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:54.8340608Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:54.8382474Z Entering 'third_party/pocketfft' 2024-06-26T07:02:54.8411574Z Entering 'third_party/protobuf' 2024-06-26T07:02:54.8443192Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:54.8470702Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:54.8500826Z Entering 'third_party/psimd' 2024-06-26T07:02:54.8529716Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:54.8558079Z Entering 'third_party/pybind11' 2024-06-26T07:02:54.8585297Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:54.8612390Z Entering 'third_party/sleef' 2024-06-26T07:02:54.8641182Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:54.8667980Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:54.8693821Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:54.8720754Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:54.8746339Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:54.8773834Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:54.8817306Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:02:54.9030591Z Entering 'android/libs/fbjni' 2024-06-26T07:02:54.9060410Z Entering 'third_party/FP16' 2024-06-26T07:02:54.9090311Z Entering 'third_party/FXdiv' 2024-06-26T07:02:54.9118405Z Entering 'third_party/NNPACK' 2024-06-26T07:02:54.9145818Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:54.9173773Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:54.9213633Z Entering 'third_party/benchmark' 2024-06-26T07:02:54.9245505Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:54.9274887Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:54.9303764Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:54.9333334Z Entering 'third_party/cutlass' 2024-06-26T07:02:54.9365633Z Entering 'third_party/eigen' 2024-06-26T07:02:54.9396656Z Entering 'third_party/fbgemm' 2024-06-26T07:02:54.9425769Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:54.9454909Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:54.9483788Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:54.9515564Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:54.9544042Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:54.9571724Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:54.9603738Z Entering 'third_party/fmt' 2024-06-26T07:02:54.9630055Z Entering 'third_party/foxi' 2024-06-26T07:02:54.9659836Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:54.9688083Z Entering 'third_party/gloo' 2024-06-26T07:02:54.9716969Z Entering 'third_party/googletest' 2024-06-26T07:02:54.9744692Z Entering 'third_party/ideep' 2024-06-26T07:02:54.9770929Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:54.9804808Z Entering 'third_party/ittapi' 2024-06-26T07:02:54.9832328Z Entering 'third_party/kineto' 2024-06-26T07:02:54.9859312Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:54.9888083Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:54.9920041Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:54.9945906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:54.9976367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:55.0002581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:55.0032968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:55.0062812Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:55.0090075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:55.0121393Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:55.0147648Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:55.0173774Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:55.0204455Z Entering 'third_party/mimalloc' 2024-06-26T07:02:55.0232097Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:55.0262210Z Entering 'third_party/nlohmann' 2024-06-26T07:02:55.0289915Z Entering 'third_party/onnx' 2024-06-26T07:02:55.0330817Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:55.0361288Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:55.0390358Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:55.0421221Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:55.0448312Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:55.0476719Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:55.0502837Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:55.0533031Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:55.0559446Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:55.0586882Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:55.0614935Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:55.0641109Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:55.0669109Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:55.0713248Z Entering 'third_party/pocketfft' 2024-06-26T07:02:55.0741038Z Entering 'third_party/protobuf' 2024-06-26T07:02:55.0773727Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:55.0799650Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:55.0828341Z Entering 'third_party/psimd' 2024-06-26T07:02:55.0855129Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:55.0884118Z Entering 'third_party/pybind11' 2024-06-26T07:02:55.0912786Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:55.0941649Z Entering 'third_party/sleef' 2024-06-26T07:02:55.0971536Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:55.1000562Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:55.1029218Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:55.1060017Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:55.1087044Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:55.1115359Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:55.1157051Z ##[endgroup] 2024-06-26T07:02:55.1190596Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:02:55.1212754Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:02:55.1393941Z Prepare all required actions 2024-06-26T07:02:55.1394775Z Getting action download info 2024-06-26T07:02:55.2776838Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:02:55.2777265Z env: 2024-06-26T07:02:55.2777544Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:55.2777891Z ##[endgroup] 2024-06-26T07:02:55.2836739Z ##[group]Run set -euo pipefail 2024-06-26T07:02:55.2837175Z set -euo pipefail 2024-06-26T07:02:55.2837563Z function get_ec2_metadata() { 2024-06-26T07:02:55.2838099Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:02:55.2838985Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:02:55.2839751Z  category=$1 2024-06-26T07:02:55.2840294Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:02:55.2840938Z  runner_name_str=i-0b34867d3f3dcf691 2024-06-26T07:02:55.2841448Z  if [[ -f /.inarc ]]; then 2024-06-26T07:02:55.2841935Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:02:55.2842502Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:02:55.2843194Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:02:55.2843799Z  else 2024-06-26T07:02:55.2844288Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:02:55.2844869Z  fi 2024-06-26T07:02:55.2845159Z } 2024-06-26T07:02:55.2845524Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:02:55.2846106Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:02:55.2846763Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:02:55.2847343Z echo "system info $(uname -a)" 2024-06-26T07:02:55.2855146Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:55.2855639Z env: 2024-06-26T07:02:55.2855924Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:55.2856280Z ##[endgroup] 2024-06-26T07:02:55.2923550Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:02:55.2962195Z instance-id: i-0b34867d3f3dcf691 2024-06-26T07:02:55.3000649Z instance-type: g5.4xlarge 2024-06-26T07:02:55.3007529Z system info Linux ip-10-0-56-211.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:02:55.3027057Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:02:55.3027996Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:02:55.3036243Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:55.3036852Z env: 2024-06-26T07:02:55.3037125Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:55.3037469Z ##[endgroup] 2024-06-26T07:02:55.3101812Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:02:55.3102515Z if systemctl is-active --quiet docker; then 2024-06-26T07:02:55.3103144Z  echo "Docker daemon is running..."; 2024-06-26T07:02:55.3103671Z else 2024-06-26T07:02:55.3104234Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:02:55.3104899Z fi 2024-06-26T07:02:55.3112117Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:55.3112690Z env: 2024-06-26T07:02:55.3113042Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:55.3113467Z ##[endgroup] 2024-06-26T07:02:55.3151334Z Docker daemon is running... 2024-06-26T07:02:55.3195550Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:02:55.3196116Z with: 2024-06-26T07:02:55.3196384Z shell: bash 2024-06-26T07:02:55.3196692Z timeout_minutes: 5 2024-06-26T07:02:55.3197018Z max_attempts: 3 2024-06-26T07:02:55.3197333Z retry_wait_seconds: 30 2024-06-26T07:02:55.3198847Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:02:55.3200266Z polling_interval_seconds: 1 2024-06-26T07:02:55.3200648Z warning_on_retry: true 2024-06-26T07:02:55.3201003Z continue_on_error: false 2024-06-26T07:02:55.3201344Z env: 2024-06-26T07:02:55.3201613Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:55.3201973Z AWS_RETRY_MODE: standard 2024-06-26T07:02:55.3202328Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:02:55.3202672Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:02:55.3203039Z ##[endgroup] 2024-06-26T07:02:56.1759288Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:56.1760382Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:56.1761457Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:56.1762011Z 2024-06-26T07:02:56.1762183Z Login Succeeded 2024-06-26T07:02:56.3703301Z Command completed after 1 attempt(s). 2024-06-26T07:02:56.3744283Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:56.3745004Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:56.3745665Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:56.3753555Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:56.3754049Z env: 2024-06-26T07:02:56.3754326Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:56.3754843Z ##[endgroup] 2024-06-26T07:02:56.3815586Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:02:56.3816454Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:02:56.3817155Z # shellcheck disable=SC2046 2024-06-26T07:02:56.3817709Z docker stop $(docker ps -q) || true 2024-06-26T07:02:56.3818288Z # Prune all of the docker images 2024-06-26T07:02:56.3818834Z docker system prune -af 2024-06-26T07:02:56.3826070Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:56.3826660Z env: 2024-06-26T07:02:56.3827014Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:56.3827450Z ##[endgroup] 2024-06-26T07:02:56.4067819Z "docker stop" requires at least 1 argument. 2024-06-26T07:02:56.4068663Z See 'docker stop --help'. 2024-06-26T07:02:56.4068941Z 2024-06-26T07:02:56.4069205Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:02:56.4069737Z 2024-06-26T07:02:56.4070081Z Stop one or more running containers 2024-06-26T07:02:56.4210797Z Total reclaimed space: 0B 2024-06-26T07:02:56.4240543Z ##[group]Run set +e 2024-06-26T07:02:56.4240896Z set +e 2024-06-26T07:02:56.4241328Z set -x 2024-06-26T07:02:56.4241621Z  2024-06-26T07:02:56.4241936Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:02:56.4242740Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:02:56.4243829Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:02:56.4244584Z # one is returned at random 2024-06-26T07:02:56.4245134Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:02:56.4245645Z  2024-06-26T07:02:56.4245971Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:02:56.4246606Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:02:56.4247381Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:02:56.4247939Z  2024-06-26T07:02:56.4248266Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:02:56.4248829Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:02:56.4249343Z  exit 1 2024-06-26T07:02:56.4249649Z  fi 2024-06-26T07:02:56.4249933Z fi 2024-06-26T07:02:56.4250316Z  2024-06-26T07:02:56.4250678Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:02:56.4251190Z  # Clean up any old records first 2024-06-26T07:02:56.4251706Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:02:56.4252166Z fi 2024-06-26T07:02:56.4252437Z  2024-06-26T07:02:56.4252872Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:02:56.4253419Z cat /etc/hosts 2024-06-26T07:02:56.4261025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:56.4261513Z env: 2024-06-26T07:02:56.4261796Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:56.4262144Z ##[endgroup] 2024-06-26T07:02:56.4281822Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:02:56.4284817Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:02:56.4285713Z ++ tail -n1 2024-06-26T07:02:56.4370887Z + RESOLVED_IP=18.160.10.76 2024-06-26T07:02:56.4371353Z + '[' -z 18.160.10.76 ']' 2024-06-26T07:02:56.4371811Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:02:56.4376731Z 18.160.10.76 download.pytorch.org 2024-06-26T07:02:56.4377834Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:02:56.4474789Z + echo '18.160.10.76 download.pytorch.org' 2024-06-26T07:02:56.4475467Z + sudo tee -a /etc/hosts 2024-06-26T07:02:56.4827877Z 18.160.10.76 download.pytorch.org 2024-06-26T07:02:56.4841409Z + cat /etc/hosts 2024-06-26T07:02:56.4846280Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:02:56.4857650Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:02:56.4858149Z 18.160.10.76 download.pytorch.org 2024-06-26T07:02:56.5061375Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:02:56.5062167Z with: 2024-06-26T07:02:56.5063269Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5064480Z docker-build-dir: .ci/docker 2024-06-26T07:02:56.5064846Z working-directory: . 2024-06-26T07:02:56.5065304Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:56.5065823Z force-push: false 2024-06-26T07:02:56.5066110Z env: 2024-06-26T07:02:56.5066374Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:56.5066708Z ##[endgroup] 2024-06-26T07:02:56.5085844Z ##[group]Run set -ex 2024-06-26T07:02:56.5086197Z set -ex 2024-06-26T07:02:56.5086485Z  2024-06-26T07:02:56.5087031Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:02:56.5087989Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:02:56.5088904Z # job could then download the pre-built image as usual 2024-06-26T07:02:56.5089631Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:02:56.5090290Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5090911Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5091453Z  2024-06-26T07:02:56.5091949Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:02:56.5092568Z  exit 0 2024-06-26T07:02:56.5092871Z else 2024-06-26T07:02:56.5093240Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5093681Z fi 2024-06-26T07:02:56.5093948Z  2024-06-26T07:02:56.5094411Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:02:56.5095252Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:02:56.5096034Z  # use it as it is, but first let's extract the tag 2024-06-26T07:02:56.5096738Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:02:56.5097467Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5098164Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5098713Z else 2024-06-26T07:02:56.5099141Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:02:56.5099797Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5100681Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5101418Z fi 2024-06-26T07:02:56.5109170Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:56.5109662Z env: 2024-06-26T07:02:56.5110091Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:56.5110454Z REPO_NAME: pytorch 2024-06-26T07:02:56.5111574Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5112757Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:02:56.5113254Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:56.5113774Z ##[endgroup] 2024-06-26T07:02:56.5133759Z + [[ ! -d .ci/docker ]] 2024-06-26T07:02:56.5134194Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:02:56.5134574Z + echo skip=false 2024-06-26T07:02:56.5136523Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:02:56.5139255Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5140442Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:02:56.5147387Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5148034Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5149714Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5176383Z ##[group]Run set +e 2024-06-26T07:02:56.5176751Z set +e 2024-06-26T07:02:56.5177059Z set -x 2024-06-26T07:02:56.5177345Z  2024-06-26T07:02:56.5177622Z login() { 2024-06-26T07:02:56.5178302Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:02:56.5179028Z } 2024-06-26T07:02:56.5179301Z  2024-06-26T07:02:56.5179695Z retry () { 2024-06-26T07:02:56.5180092Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:02:56.5180543Z } 2024-06-26T07:02:56.5180813Z  2024-06-26T07:02:56.5181137Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:02:56.5181556Z  2024-06-26T07:02:56.5182021Z # Check if image already exists, if it does then skip building it 2024-06-26T07:02:56.5182725Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:02:56.5183235Z  exit 0 2024-06-26T07:02:56.5183544Z fi 2024-06-26T07:02:56.5183817Z  2024-06-26T07:02:56.5184326Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:02:56.5185156Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:02:56.5185901Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:02:56.5186570Z  # if we're on the base branch then use the parent commit 2024-06-26T07:02:56.5187184Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:02:56.5187619Z else 2024-06-26T07:02:56.5188097Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:02:56.5188785Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:02:56.5189296Z fi 2024-06-26T07:02:56.5189579Z  2024-06-26T07:02:56.5190063Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:02:56.5190574Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5191039Z  2024-06-26T07:02:56.5191691Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:02:56.5192457Z  exit 0 2024-06-26T07:02:56.5192755Z fi 2024-06-26T07:02:56.5193032Z  2024-06-26T07:02:56.5193482Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:02:56.5194711Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:02:56.5195578Z  exit 1 2024-06-26T07:02:56.5195888Z fi 2024-06-26T07:02:56.5196173Z  2024-06-26T07:02:56.5196669Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:02:56.5197613Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:02:56.5198483Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:02:56.5199445Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:02:56.5200541Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:02:56.5201293Z fi 2024-06-26T07:02:56.5201577Z  2024-06-26T07:02:56.5201947Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:56.5209698Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:56.5210205Z env: 2024-06-26T07:02:56.5210485Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:56.5210854Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:02:56.5211321Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:56.5212579Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5213818Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:56.5214425Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:56.5214947Z ##[endgroup] 2024-06-26T07:02:56.5235054Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:56.5235918Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:56.5237022Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:02:56.5237797Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:56.9364161Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:56.9365076Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:56.9366046Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:56.9366560Z 2024-06-26T07:02:56.9366687Z Login Succeeded 2024-06-26T07:02:56.9376858Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.1489144Z { 2024-06-26T07:02:57.1489593Z "schemaVersion": 2, 2024-06-26T07:02:57.1490264Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:02:57.1490846Z "config": { 2024-06-26T07:02:57.1491381Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:02:57.1492064Z "size": 48681, 2024-06-26T07:02:57.1492771Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:02:57.1493486Z }, 2024-06-26T07:02:57.1493840Z "layers": [ 2024-06-26T07:02:57.1494210Z { 2024-06-26T07:02:57.1494674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1495229Z "size": 28580681, 2024-06-26T07:02:57.1496347Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:02:57.1497207Z }, 2024-06-26T07:02:57.1497548Z { 2024-06-26T07:02:57.1498012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1498560Z "size": 7943451, 2024-06-26T07:02:57.1499104Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:02:57.1499711Z }, 2024-06-26T07:02:57.1499956Z { 2024-06-26T07:02:57.1500646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1501433Z "size": 55728572, 2024-06-26T07:02:57.1502217Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:02:57.1503030Z }, 2024-06-26T07:02:57.1503354Z { 2024-06-26T07:02:57.1503926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1504661Z "size": 186, 2024-06-26T07:02:57.1505254Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:02:57.1505884Z }, 2024-06-26T07:02:57.1506129Z { 2024-06-26T07:02:57.1506558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1507195Z "size": 6886, 2024-06-26T07:02:57.1507915Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:02:57.1508730Z }, 2024-06-26T07:02:57.1509354Z { 2024-06-26T07:02:57.1510099Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1510985Z + exit 0 2024-06-26T07:02:57.1511265Z "size": 1329001756, 2024-06-26T07:02:57.1511841Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:02:57.1512464Z }, 2024-06-26T07:02:57.1512713Z { 2024-06-26T07:02:57.1513129Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1513675Z "size": 62501, 2024-06-26T07:02:57.1514215Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:02:57.1515329Z }, 2024-06-26T07:02:57.1515702Z { 2024-06-26T07:02:57.1516168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1516926Z "size": 1684, 2024-06-26T07:02:57.1517637Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:02:57.1518453Z }, 2024-06-26T07:02:57.1518760Z { 2024-06-26T07:02:57.1519339Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1520293Z "size": 1523, 2024-06-26T07:02:57.1521034Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:02:57.1521868Z }, 2024-06-26T07:02:57.1522166Z { 2024-06-26T07:02:57.1522598Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1523154Z "size": 2528295403, 2024-06-26T07:02:57.1523719Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:02:57.1524326Z }, 2024-06-26T07:02:57.1524570Z { 2024-06-26T07:02:57.1525002Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1525547Z "size": 86016, 2024-06-26T07:02:57.1526094Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:02:57.1526729Z }, 2024-06-26T07:02:57.1526972Z { 2024-06-26T07:02:57.1527407Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1527958Z "size": 1823, 2024-06-26T07:02:57.1528493Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:02:57.1529099Z }, 2024-06-26T07:02:57.1529337Z { 2024-06-26T07:02:57.1529772Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1530328Z "size": 246737542, 2024-06-26T07:02:57.1530891Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:02:57.1531489Z }, 2024-06-26T07:02:57.1531733Z { 2024-06-26T07:02:57.1532161Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1532716Z "size": 545, 2024-06-26T07:02:57.1533257Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:02:57.1533878Z }, 2024-06-26T07:02:57.1534124Z { 2024-06-26T07:02:57.1534555Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1535110Z "size": 1283, 2024-06-26T07:02:57.1535645Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:02:57.1536263Z }, 2024-06-26T07:02:57.1536507Z { 2024-06-26T07:02:57.1536944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1537501Z "size": 485, 2024-06-26T07:02:57.1538056Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:02:57.1538681Z }, 2024-06-26T07:02:57.1538937Z { 2024-06-26T07:02:57.1539371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1539922Z "size": 91710937, 2024-06-26T07:02:57.1540490Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:02:57.1541128Z }, 2024-06-26T07:02:57.1541378Z { 2024-06-26T07:02:57.1541801Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1542490Z "size": 3233, 2024-06-26T07:02:57.1543050Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:02:57.1543676Z }, 2024-06-26T07:02:57.1543912Z { 2024-06-26T07:02:57.1544340Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1544899Z "size": 1905, 2024-06-26T07:02:57.1545450Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:02:57.1546097Z }, 2024-06-26T07:02:57.1546366Z { 2024-06-26T07:02:57.1546803Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1547360Z "size": 702, 2024-06-26T07:02:57.1547891Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:57.1548515Z }, 2024-06-26T07:02:57.1548766Z { 2024-06-26T07:02:57.1549188Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1549746Z "size": 2784789002, 2024-06-26T07:02:57.1550444Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:02:57.1551106Z }, 2024-06-26T07:02:57.1551343Z { 2024-06-26T07:02:57.1551777Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1552323Z "size": 381, 2024-06-26T07:02:57.1552866Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:02:57.1553462Z }, 2024-06-26T07:02:57.1553712Z { 2024-06-26T07:02:57.1554138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1554878Z "size": 11879, 2024-06-26T07:02:57.1555423Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:02:57.1556020Z }, 2024-06-26T07:02:57.1556269Z { 2024-06-26T07:02:57.1556685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1557232Z "size": 803, 2024-06-26T07:02:57.1557778Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:02:57.1558407Z }, 2024-06-26T07:02:57.1558651Z { 2024-06-26T07:02:57.1559081Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1559627Z "size": 106, 2024-06-26T07:02:57.1560158Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:02:57.1560743Z }, 2024-06-26T07:02:57.1560992Z { 2024-06-26T07:02:57.1561425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1561962Z "size": 503, 2024-06-26T07:02:57.1562498Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:02:57.1563104Z }, 2024-06-26T07:02:57.1563347Z { 2024-06-26T07:02:57.1563760Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1564307Z "size": 121477177, 2024-06-26T07:02:57.1564871Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:02:57.1565473Z }, 2024-06-26T07:02:57.1565714Z { 2024-06-26T07:02:57.1566150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1566694Z "size": 109, 2024-06-26T07:02:57.1567221Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:02:57.1567829Z }, 2024-06-26T07:02:57.1568078Z { 2024-06-26T07:02:57.1568513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1569049Z "size": 491, 2024-06-26T07:02:57.1569574Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:02:57.1570172Z }, 2024-06-26T07:02:57.1570429Z { 2024-06-26T07:02:57.1570850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1571404Z "size": 291, 2024-06-26T07:02:57.1571961Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:02:57.1572672Z }, 2024-06-26T07:02:57.1572922Z { 2024-06-26T07:02:57.1573375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1573937Z "size": 101, 2024-06-26T07:02:57.1574472Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:02:57.1575082Z }, 2024-06-26T07:02:57.1575332Z { 2024-06-26T07:02:57.1575811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1576352Z "size": 1473, 2024-06-26T07:02:57.1576889Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:02:57.1577498Z }, 2024-06-26T07:02:57.1577749Z { 2024-06-26T07:02:57.1578166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1578715Z "size": 446710781, 2024-06-26T07:02:57.1579281Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:02:57.1579884Z }, 2024-06-26T07:02:57.1580159Z { 2024-06-26T07:02:57.1580593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1581220Z "size": 161, 2024-06-26T07:02:57.1581732Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:02:57.1582328Z }, 2024-06-26T07:02:57.1582575Z { 2024-06-26T07:02:57.1583006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1583544Z "size": 564, 2024-06-26T07:02:57.1584091Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:02:57.1584703Z }, 2024-06-26T07:02:57.1584952Z { 2024-06-26T07:02:57.1585379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1585934Z "size": 35874600, 2024-06-26T07:02:57.1586485Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:02:57.1587086Z }, 2024-06-26T07:02:57.1587335Z { 2024-06-26T07:02:57.1587769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1588330Z "size": 104, 2024-06-26T07:02:57.1588840Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:02:57.1589434Z }, 2024-06-26T07:02:57.1589677Z { 2024-06-26T07:02:57.1590216Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1590754Z "size": 425, 2024-06-26T07:02:57.1591283Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:02:57.1591882Z }, 2024-06-26T07:02:57.1592123Z { 2024-06-26T07:02:57.1592555Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1593109Z "size": 20262084, 2024-06-26T07:02:57.1593684Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:02:57.1594294Z }, 2024-06-26T07:02:57.1594722Z { 2024-06-26T07:02:57.1595169Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1595769Z "size": 439, 2024-06-26T07:02:57.1596298Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:02:57.1596906Z }, 2024-06-26T07:02:57.1597160Z { 2024-06-26T07:02:57.1597596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1598140Z "size": 702, 2024-06-26T07:02:57.1598695Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:57.1599315Z }, 2024-06-26T07:02:57.1599555Z { 2024-06-26T07:02:57.1599991Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1600551Z "size": 143, 2024-06-26T07:02:57.1601087Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:02:57.1601674Z }, 2024-06-26T07:02:57.1601921Z { 2024-06-26T07:02:57.1602354Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1602999Z "size": 135, 2024-06-26T07:02:57.1603532Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:02:57.1604163Z }, 2024-06-26T07:02:57.1604414Z { 2024-06-26T07:02:57.1604833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1605411Z "size": 4911495247, 2024-06-26T07:02:57.1605991Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:02:57.1606582Z }, 2024-06-26T07:02:57.1606826Z { 2024-06-26T07:02:57.1607253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1607797Z "size": 190, 2024-06-26T07:02:57.1608342Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:02:57.1608955Z }, 2024-06-26T07:02:57.1609204Z { 2024-06-26T07:02:57.1609635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1610182Z "size": 565, 2024-06-26T07:02:57.1610706Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:02:57.1611376Z }, 2024-06-26T07:02:57.1611623Z { 2024-06-26T07:02:57.1612042Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1612590Z "size": 43163095, 2024-06-26T07:02:57.1613146Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:02:57.1613756Z }, 2024-06-26T07:02:57.1613997Z { 2024-06-26T07:02:57.1614435Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1614983Z "size": 106, 2024-06-26T07:02:57.1615530Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:02:57.1616144Z }, 2024-06-26T07:02:57.1616404Z { 2024-06-26T07:02:57.1616836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1617488Z "size": 1213, 2024-06-26T07:02:57.1618032Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:02:57.1618647Z }, 2024-06-26T07:02:57.1618888Z { 2024-06-26T07:02:57.1619327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1619882Z "size": 702, 2024-06-26T07:02:57.1620416Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:57.1621034Z }, 2024-06-26T07:02:57.1621290Z { 2024-06-26T07:02:57.1621733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1622278Z "size": 138, 2024-06-26T07:02:57.1622821Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:02:57.1623457Z }, 2024-06-26T07:02:57.1623717Z { 2024-06-26T07:02:57.1624138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1624679Z "size": 120, 2024-06-26T07:02:57.1625219Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:02:57.1625829Z }, 2024-06-26T07:02:57.1626079Z { 2024-06-26T07:02:57.1626511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1627064Z "size": 1904157495, 2024-06-26T07:02:57.1627620Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:02:57.1628238Z }, 2024-06-26T07:02:57.1628482Z { 2024-06-26T07:02:57.1628908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1629440Z "size": 172, 2024-06-26T07:02:57.1630094Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:02:57.1630715Z }, 2024-06-26T07:02:57.1630967Z { 2024-06-26T07:02:57.1631383Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1631923Z "size": 908, 2024-06-26T07:02:57.1632471Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:02:57.1633149Z }, 2024-06-26T07:02:57.1633395Z { 2024-06-26T07:02:57.1633826Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1634544Z "size": 702, 2024-06-26T07:02:57.1635100Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:57.1635707Z }, 2024-06-26T07:02:57.1635953Z { 2024-06-26T07:02:57.1636379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1636915Z "size": 134, 2024-06-26T07:02:57.1637458Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:02:57.1638063Z }, 2024-06-26T07:02:57.1638306Z { 2024-06-26T07:02:57.1638738Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1639285Z "size": 32, 2024-06-26T07:02:57.1639817Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:57.1640417Z }, 2024-06-26T07:02:57.1640678Z { 2024-06-26T07:02:57.1641105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1641749Z "size": 156, 2024-06-26T07:02:57.1642290Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:02:57.1642909Z }, 2024-06-26T07:02:57.1643157Z { 2024-06-26T07:02:57.1643582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1644122Z "size": 1841, 2024-06-26T07:02:57.1644665Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:02:57.1645279Z }, 2024-06-26T07:02:57.1645520Z { 2024-06-26T07:02:57.1645954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1646503Z "size": 7529784, 2024-06-26T07:02:57.1647067Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:02:57.1647676Z }, 2024-06-26T07:02:57.1647937Z { 2024-06-26T07:02:57.1648385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1648952Z "size": 165, 2024-06-26T07:02:57.1649495Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:02:57.1650117Z }, 2024-06-26T07:02:57.1650371Z { 2024-06-26T07:02:57.1650800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1651361Z "size": 7944, 2024-06-26T07:02:57.1651908Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:02:57.1652519Z }, 2024-06-26T07:02:57.1652758Z { 2024-06-26T07:02:57.1653188Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1653744Z "size": 8070, 2024-06-26T07:02:57.1654287Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:02:57.1654886Z }, 2024-06-26T07:02:57.1655134Z { 2024-06-26T07:02:57.1655573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1656132Z "size": 303, 2024-06-26T07:02:57.1656682Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:02:57.1657293Z }, 2024-06-26T07:02:57.1657535Z { 2024-06-26T07:02:57.1657957Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1658508Z "size": 7629071, 2024-06-26T07:02:57.1659030Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:02:57.1659614Z }, 2024-06-26T07:02:57.1659852Z { 2024-06-26T07:02:57.1660273Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1660811Z "size": 108, 2024-06-26T07:02:57.1661346Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:02:57.1661947Z }, 2024-06-26T07:02:57.1662191Z { 2024-06-26T07:02:57.1662612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1663225Z "size": 54145775, 2024-06-26T07:02:57.1663780Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:02:57.1664405Z }, 2024-06-26T07:02:57.1664652Z { 2024-06-26T07:02:57.1665070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1665610Z "size": 474, 2024-06-26T07:02:57.1666158Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:02:57.1666773Z }, 2024-06-26T07:02:57.1667013Z { 2024-06-26T07:02:57.1667445Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1667994Z "size": 1374858747, 2024-06-26T07:02:57.1668539Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:02:57.1669125Z }, 2024-06-26T07:02:57.1669378Z { 2024-06-26T07:02:57.1669802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1670447Z "size": 106, 2024-06-26T07:02:57.1670998Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:02:57.1671672Z }, 2024-06-26T07:02:57.1671919Z { 2024-06-26T07:02:57.1672338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1672882Z "size": 558, 2024-06-26T07:02:57.1673424Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:02:57.1674036Z }, 2024-06-26T07:02:57.1674272Z { 2024-06-26T07:02:57.1674900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1675459Z "size": 46248513, 2024-06-26T07:02:57.1675999Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:02:57.1676606Z }, 2024-06-26T07:02:57.1676856Z { 2024-06-26T07:02:57.1677284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1677819Z "size": 111, 2024-06-26T07:02:57.1678360Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:02:57.1678964Z }, 2024-06-26T07:02:57.1679205Z { 2024-06-26T07:02:57.1679622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1680161Z "size": 32, 2024-06-26T07:02:57.1680690Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:57.1681292Z }, 2024-06-26T07:02:57.1681526Z { 2024-06-26T07:02:57.1681948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1682488Z "size": 32, 2024-06-26T07:02:57.1683008Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:57.1683612Z }, 2024-06-26T07:02:57.1683855Z { 2024-06-26T07:02:57.1684279Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1684808Z "size": 32, 2024-06-26T07:02:57.1685341Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:57.1685945Z }, 2024-06-26T07:02:57.1686185Z { 2024-06-26T07:02:57.1686609Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:57.1687147Z "size": 32, 2024-06-26T07:02:57.1687677Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:57.1688284Z } 2024-06-26T07:02:57.1688520Z ] 2024-06-26T07:02:57.1688764Z } 2024-06-26T07:02:57.1802558Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:02:57.1803050Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:02:57.1803591Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:02:57.1811440Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:57.1811941Z env: 2024-06-26T07:02:57.1812218Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:57.1813354Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.1814513Z ##[endgroup] 2024-06-26T07:02:57.1839832Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.1889367Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:02:57.1889943Z with: 2024-06-26T07:02:57.1891004Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.1892309Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:57.1892822Z env: 2024-06-26T07:02:57.1893105Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:57.1893463Z ##[endgroup] 2024-06-26T07:02:57.1908812Z ##[group]Run set -x 2024-06-26T07:02:57.1909156Z set -x 2024-06-26T07:02:57.1909464Z set +e 2024-06-26T07:02:57.1909760Z  2024-06-26T07:02:57.1910153Z login() { 2024-06-26T07:02:57.1910828Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:02:57.1911673Z } 2024-06-26T07:02:57.1911959Z  2024-06-26T07:02:57.1912278Z retry () { 2024-06-26T07:02:57.1912679Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:02:57.1913131Z } 2024-06-26T07:02:57.1913410Z  2024-06-26T07:02:57.1913735Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:02:57.1914147Z  2024-06-26T07:02:57.1914562Z set -e 2024-06-26T07:02:57.1915038Z # ignore output since only exit code is used for conditional 2024-06-26T07:02:57.1915783Z # only pull docker image if it's not available locally 2024-06-26T07:02:57.1916536Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:02:57.1917231Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:02:57.1917670Z fi 2024-06-26T07:02:57.1925132Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:57.1925629Z env: 2024-06-26T07:02:57.1925907Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:57.1927032Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.1928323Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:57.1928833Z ##[endgroup] 2024-06-26T07:02:57.1949468Z + set +e 2024-06-26T07:02:57.1950231Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:57.1950870Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:57.1951577Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:02:57.1952800Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:57.6035928Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:57.6036875Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:57.6037846Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:57.6038374Z 2024-06-26T07:02:57.6038495Z Login Succeeded 2024-06-26T07:02:57.6046877Z + set -e 2024-06-26T07:02:57.6048233Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.6176668Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.6178819Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:57.8417555Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:02:57.8418600Z 7a2c55901189: Pulling fs layer 2024-06-26T07:02:57.8420695Z 224fe954d725: Pulling fs layer 2024-06-26T07:02:57.8421332Z 75722010b82e: Pulling fs layer 2024-06-26T07:02:57.8421798Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:02:57.8422184Z b57676e46aee: Pulling fs layer 2024-06-26T07:02:57.8422630Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:02:57.8423015Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:02:57.8423386Z 0c1227890755: Pulling fs layer 2024-06-26T07:02:57.8423758Z d8d1234baab3: Pulling fs layer 2024-06-26T07:02:57.8424127Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:02:57.8424500Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:02:57.8424874Z 309b872da805: Pulling fs layer 2024-06-26T07:02:57.8425245Z e82932cb002c: Pulling fs layer 2024-06-26T07:02:57.8425635Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:02:57.8426087Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:02:57.8426461Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:02:57.8426829Z f4863801e30c: Pulling fs layer 2024-06-26T07:02:57.8427303Z a8c1e85b5e14: Waiting 2024-06-26T07:02:57.8427631Z a41a8d1c11c8: Waiting 2024-06-26T07:02:57.8427977Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:02:57.8428353Z cf2f34f225db: Pulling fs layer 2024-06-26T07:02:57.8428704Z 0c1227890755: Waiting 2024-06-26T07:02:57.8429038Z df546d7b3e34: Pulling fs layer 2024-06-26T07:02:57.8429411Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:02:57.8429779Z 96d0e2032184: Pulling fs layer 2024-06-26T07:02:57.8430274Z 06e12ce59790: Pulling fs layer 2024-06-26T07:02:57.8430720Z ed712f63cde7: Pulling fs layer 2024-06-26T07:02:57.8431104Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:02:57.8431536Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:02:57.8431917Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:02:57.8432347Z de45a14c9b22: Pulling fs layer 2024-06-26T07:02:57.8432724Z e99241e77209: Pulling fs layer 2024-06-26T07:02:57.8433151Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:02:57.8433528Z a823ffa4c541: Pulling fs layer 2024-06-26T07:02:57.8433964Z 44ba272eae20: Pulling fs layer 2024-06-26T07:02:57.8434524Z da02d278e524: Pulling fs layer 2024-06-26T07:02:57.8434929Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:02:57.8435296Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:02:57.8435666Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:02:57.8436021Z d8d1234baab3: Waiting 2024-06-26T07:02:57.8436352Z b2950a036534: Pulling fs layer 2024-06-26T07:02:57.8436724Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:02:57.8437091Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:02:57.8437442Z d37dd9ed8948: Waiting 2024-06-26T07:02:57.8437773Z 7a868239227f: Pulling fs layer 2024-06-26T07:02:57.8438132Z 7ed32bc8e469: Waiting 2024-06-26T07:02:57.8438463Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:02:57.8438830Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:02:57.8439197Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:02:57.8439555Z cf2f34f225db: Waiting 2024-06-26T07:02:57.8439951Z c1068d03a811: Pulling fs layer 2024-06-26T07:02:57.8440328Z 671b4a301d81: Pulling fs layer 2024-06-26T07:02:57.8440761Z ec1e7978c1fe: Waiting 2024-06-26T07:02:57.8441153Z df546d7b3e34: Waiting 2024-06-26T07:02:57.8441492Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:02:57.8441917Z a97a71ef8a4e: Waiting 2024-06-26T07:02:57.8442298Z 06e12ce59790: Waiting 2024-06-26T07:02:57.8442629Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:02:57.8443063Z fb2e97768560: Pulling fs layer 2024-06-26T07:02:57.8443432Z 309b872da805: Waiting 2024-06-26T07:02:57.8443742Z 25cd0c482ad6: Waiting 2024-06-26T07:02:57.8444052Z ed712f63cde7: Waiting 2024-06-26T07:02:57.8444365Z e82932cb002c: Waiting 2024-06-26T07:02:57.8444672Z 96d0e2032184: Waiting 2024-06-26T07:02:57.8444984Z d527cbbb87e3: Waiting 2024-06-26T07:02:57.8445289Z 3d914950f8ad: Waiting 2024-06-26T07:02:57.8445624Z 9bf94bc45658: Waiting 2024-06-26T07:02:57.8445960Z de45a14c9b22: Waiting 2024-06-26T07:02:57.8446268Z ca9dd73e3e93: Waiting 2024-06-26T07:02:57.8446582Z 44ba272eae20: Waiting 2024-06-26T07:02:57.8446891Z c1068d03a811: Waiting 2024-06-26T07:02:57.8447208Z b2950a036534: Waiting 2024-06-26T07:02:57.8447509Z e99241e77209: Waiting 2024-06-26T07:02:57.8447817Z 16d76a79d5d6: Waiting 2024-06-26T07:02:57.8448259Z 84b2c7a89982: Waiting 2024-06-26T07:02:57.8448581Z 25e19d81d9fc: Waiting 2024-06-26T07:02:57.8448980Z b3c5d66c679f: Waiting 2024-06-26T07:02:57.8449295Z 83911d7c7ecf: Waiting 2024-06-26T07:02:57.8449648Z 49737bd5ae70: Waiting 2024-06-26T07:02:57.8449985Z 35e8f4e2a42f: Waiting 2024-06-26T07:02:57.8450315Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:02:57.8450760Z e08e88c13668: Pulling fs layer 2024-06-26T07:02:57.8451135Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:02:57.8451554Z a823ffa4c541: Waiting 2024-06-26T07:02:57.8451873Z 60156b40fcbe: Waiting 2024-06-26T07:02:57.8452227Z 7a868239227f: Waiting 2024-06-26T07:02:57.8452562Z 67760cf5ec46: Waiting 2024-06-26T07:02:57.8452873Z fb2e97768560: Waiting 2024-06-26T07:02:57.8453251Z ddc9740bcea2: Waiting 2024-06-26T07:02:57.8453569Z da02d278e524: Waiting 2024-06-26T07:02:57.8453901Z 73a0ce681c29: Waiting 2024-06-26T07:02:57.8454366Z 4bdf41c9ecbe: Waiting 2024-06-26T07:02:57.8454693Z 12ba7431ff38: Waiting 2024-06-26T07:02:57.8455082Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:02:57.8455474Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:02:57.8455904Z cf7c2cb896cb: Waiting 2024-06-26T07:02:57.8456229Z 3a778ddfac2a: Waiting 2024-06-26T07:02:57.8456618Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:02:57.8456998Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:02:57.8457433Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:02:57.8457834Z e797fe86caef: Pulling fs layer 2024-06-26T07:02:57.8458227Z 967190076fc0: Pulling fs layer 2024-06-26T07:02:57.8458626Z dda1ce302b9a: Waiting 2024-06-26T07:02:57.8458958Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:02:57.8459375Z 236c8f0245f4: Waiting 2024-06-26T07:02:57.8459694Z 967190076fc0: Waiting 2024-06-26T07:02:57.8460094Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:02:57.8460463Z e797fe86caef: Waiting 2024-06-26T07:02:57.8460826Z cc91933e985a: Pulling fs layer 2024-06-26T07:02:57.8461226Z 4fb0a83f0685: Waiting 2024-06-26T07:02:57.8461566Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:02:57.8461995Z 526083994857: Pulling fs layer 2024-06-26T07:02:57.8462368Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:02:57.8462809Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:02:57.8463181Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:02:57.8463607Z 95237e7311a1: Pulling fs layer 2024-06-26T07:02:57.8463979Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:02:57.8464417Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:02:57.8464784Z 68af2ec4e30a: Waiting 2024-06-26T07:02:57.8465157Z 2060028f7488: Pulling fs layer 2024-06-26T07:02:57.8465569Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:02:57.8465998Z cc91933e985a: Waiting 2024-06-26T07:02:57.8466331Z 526083994857: Waiting 2024-06-26T07:02:57.8466634Z 2060028f7488: Waiting 2024-06-26T07:02:57.8467007Z 9f3187916e7d: Waiting 2024-06-26T07:02:57.8467327Z e412c7f3b2de: Waiting 2024-06-26T07:02:57.8467673Z 789eff6c51aa: Waiting 2024-06-26T07:02:57.8468019Z 05a6f6dcc1c0: Waiting 2024-06-26T07:02:57.8468342Z 6bc9dc8315b0: Waiting 2024-06-26T07:02:57.8468717Z 95237e7311a1: Waiting 2024-06-26T07:02:57.8469030Z de0f8a8c45c6: Waiting 2024-06-26T07:02:57.9798585Z 224fe954d725: Verifying Checksum 2024-06-26T07:02:57.9799072Z 224fe954d725: Download complete 2024-06-26T07:02:58.0649071Z d527cbbb87e3: Download complete 2024-06-26T07:02:58.1798763Z b57676e46aee: Verifying Checksum 2024-06-26T07:02:58.1799235Z b57676e46aee: Download complete 2024-06-26T07:02:58.1864937Z 7a2c55901189: Verifying Checksum 2024-06-26T07:02:58.1865341Z 7a2c55901189: Download complete 2024-06-26T07:02:58.2972725Z a41a8d1c11c8: Verifying Checksum 2024-06-26T07:02:58.2973161Z a41a8d1c11c8: Download complete 2024-06-26T07:02:58.3665082Z 0c1227890755: Verifying Checksum 2024-06-26T07:02:58.3665665Z 0c1227890755: Download complete 2024-06-26T07:02:58.4484283Z d8d1234baab3: Verifying Checksum 2024-06-26T07:02:58.4484771Z d8d1234baab3: Download complete 2024-06-26T07:02:58.4602224Z 75722010b82e: Verifying Checksum 2024-06-26T07:02:58.4602832Z 75722010b82e: Download complete 2024-06-26T07:02:58.5368597Z ec1e7978c1fe: Download complete 2024-06-26T07:02:58.6092165Z 309b872da805: Verifying Checksum 2024-06-26T07:02:58.6092690Z 309b872da805: Download complete 2024-06-26T07:02:59.2169581Z 7a2c55901189: Pull complete 2024-06-26T07:02:59.5189956Z 224fe954d725: Pull complete 2024-06-26T07:03:00.4319834Z 75722010b82e: Pull complete 2024-06-26T07:03:00.4765568Z d527cbbb87e3: Pull complete 2024-06-26T07:03:00.5283505Z b57676e46aee: Pull complete 2024-06-26T07:03:01.1239289Z e82932cb002c: Verifying Checksum 2024-06-26T07:03:01.1239816Z e82932cb002c: Download complete 2024-06-26T07:03:01.2126597Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:03:01.2127228Z ca9dd73e3e93: Download complete 2024-06-26T07:03:01.2851095Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:03:01.2851531Z 16d76a79d5d6: Download complete 2024-06-26T07:03:01.3729982Z 4bdf41c9ecbe: Download complete 2024-06-26T07:03:02.3533362Z f4863801e30c: Verifying Checksum 2024-06-26T07:03:02.3534066Z f4863801e30c: Download complete 2024-06-26T07:03:02.4409697Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:03:02.4410202Z d37dd9ed8948: Download complete 2024-06-26T07:03:02.5220007Z cf2f34f225db: Verifying Checksum 2024-06-26T07:03:02.5220640Z cf2f34f225db: Download complete 2024-06-26T07:03:02.5974153Z df546d7b3e34: Verifying Checksum 2024-06-26T07:03:02.5974635Z df546d7b3e34: Download complete 2024-06-26T07:03:11.5195198Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:03:11.5195669Z a8c1e85b5e14: Download complete 2024-06-26T07:03:11.5920338Z 96d0e2032184: Download complete 2024-06-26T07:03:11.6728016Z 06e12ce59790: Verifying Checksum 2024-06-26T07:03:11.6728523Z 06e12ce59790: Download complete 2024-06-26T07:03:11.7458439Z ed712f63cde7: Download complete 2024-06-26T07:03:11.8256848Z a97a71ef8a4e: Download complete 2024-06-26T07:03:11.9047108Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:03:11.9047675Z 25cd0c482ad6: Download complete 2024-06-26T07:03:13.1636368Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:03:13.1636868Z 9bf94bc45658: Download complete 2024-06-26T07:03:13.2389282Z de45a14c9b22: Verifying Checksum 2024-06-26T07:03:13.2389841Z de45a14c9b22: Download complete 2024-06-26T07:03:13.3060695Z e99241e77209: Download complete 2024-06-26T07:03:13.3651776Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:03:13.3652227Z 35e8f4e2a42f: Download complete 2024-06-26T07:03:13.4387796Z a823ffa4c541: Verifying Checksum 2024-06-26T07:03:13.4388372Z a823ffa4c541: Download complete 2024-06-26T07:03:13.5184648Z 44ba272eae20: Download complete 2024-06-26T07:03:18.0412723Z da02d278e524: Verifying Checksum 2024-06-26T07:03:18.0413323Z da02d278e524: Download complete 2024-06-26T07:03:18.1349759Z 73a0ce681c29: Download complete 2024-06-26T07:03:18.2462339Z 49737bd5ae70: Download complete 2024-06-26T07:03:18.6669460Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:03:18.6670183Z 9cd4c745f8e4: Download complete 2024-06-26T07:03:18.7498427Z b2950a036534: Download complete 2024-06-26T07:03:18.8463888Z 25e19d81d9fc: Download complete 2024-06-26T07:03:19.2814667Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:03:19.2815205Z 60156b40fcbe: Download complete 2024-06-26T07:03:19.3553230Z 7a868239227f: Download complete 2024-06-26T07:03:19.4280978Z 83911d7c7ecf: Download complete 2024-06-26T07:03:19.5059146Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:03:23.8647428Z b3c5d66c679f: Download complete 2024-06-26T07:03:23.8648092Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:03:23.8648709Z 7ed32bc8e469: Download complete 2024-06-26T07:03:23.9337720Z c1068d03a811: Verifying Checksum 2024-06-26T07:03:23.9338269Z c1068d03a811: Download complete 2024-06-26T07:03:24.0498353Z 671b4a301d81: Verifying Checksum 2024-06-26T07:03:24.0499065Z 671b4a301d81: Download complete 2024-06-26T07:03:24.2665100Z a8c1e85b5e14: Pull complete 2024-06-26T07:03:24.5108929Z a41a8d1c11c8: Pull complete 2024-06-26T07:03:24.5408264Z 67760cf5ec46: Download complete 2024-06-26T07:03:24.6152539Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:03:24.7314535Z 0c1227890755: Pull complete 2024-06-26T07:03:24.7634312Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:03:24.7635075Z b599c5a8c7cb: Download complete 2024-06-26T07:03:24.8380960Z e08e88c13668: Verifying Checksum 2024-06-26T07:03:24.8381459Z e08e88c13668: Download complete 2024-06-26T07:03:24.9660045Z d8d1234baab3: Pull complete 2024-06-26T07:03:44.9333772Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:03:44.9334269Z 3d914950f8ad: Download complete 2024-06-26T07:03:45.0237443Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:03:45.0237907Z cf7c2cb896cb: Download complete 2024-06-26T07:03:45.1082981Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:03:45.1083487Z 3a778ddfac2a: Download complete 2024-06-26T07:03:45.1838022Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:03:45.1838489Z 236c8f0245f4: Download complete 2024-06-26T07:03:45.1939855Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:03:45.1940270Z 4f4fb700ef54: Download complete 2024-06-26T07:03:45.2809088Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:03:45.2809953Z dda1ce302b9a: Download complete 2024-06-26T07:03:45.3479338Z e797fe86caef: Verifying Checksum 2024-06-26T07:03:45.3479902Z e797fe86caef: Download complete 2024-06-26T07:03:45.7024745Z 967190076fc0: Verifying Checksum 2024-06-26T07:03:45.7025272Z 967190076fc0: Download complete 2024-06-26T07:03:45.7851269Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:03:45.7851847Z 4fb0a83f0685: Download complete 2024-06-26T07:03:45.8696739Z 68af2ec4e30a: Download complete 2024-06-26T07:03:45.9933795Z cc91933e985a: Verifying Checksum 2024-06-26T07:03:45.9934251Z cc91933e985a: Download complete 2024-06-26T07:03:46.0709326Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:03:46.0709771Z 9f3187916e7d: Download complete 2024-06-26T07:03:46.4236469Z 526083994857: Verifying Checksum 2024-06-26T07:03:46.4236910Z 526083994857: Download complete 2024-06-26T07:03:46.4923958Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:03:46.4924376Z 05a6f6dcc1c0: Download complete 2024-06-26T07:03:48.5340191Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:03:48.5340705Z 6bc9dc8315b0: Download complete 2024-06-26T07:03:48.6497185Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:03:48.6498059Z 1b0e27c35f07: Download complete 2024-06-26T07:03:52.9585871Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:03:52.9586397Z 12ba7431ff38: Download complete 2024-06-26T07:03:53.0461066Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:03:53.0461619Z 789eff6c51aa: Download complete 2024-06-26T07:03:53.1296422Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:03:53.1296931Z de0f8a8c45c6: Download complete 2024-06-26T07:03:55.0043901Z 2060028f7488: Verifying Checksum 2024-06-26T07:03:55.0044549Z 2060028f7488: Download complete 2024-06-26T07:03:55.0863958Z e412c7f3b2de: Download complete 2024-06-26T07:04:17.4853788Z 95237e7311a1: Verifying Checksum 2024-06-26T07:04:17.4854269Z 95237e7311a1: Download complete 2024-06-26T07:04:36.9274149Z 7ed32bc8e469: Pull complete 2024-06-26T07:04:37.2232306Z ec1e7978c1fe: Pull complete 2024-06-26T07:04:37.4778388Z 309b872da805: Pull complete 2024-06-26T07:04:45.0185381Z e82932cb002c: Pull complete 2024-06-26T07:04:45.2378598Z ca9dd73e3e93: Pull complete 2024-06-26T07:04:45.4777753Z 16d76a79d5d6: Pull complete 2024-06-26T07:04:45.6246866Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:04:47.9094589Z f4863801e30c: Pull complete 2024-06-26T07:04:48.1111410Z d37dd9ed8948: Pull complete 2024-06-26T07:04:48.3383275Z cf2f34f225db: Pull complete 2024-06-26T07:04:48.5390897Z df546d7b3e34: Pull complete 2024-06-26T07:04:49.8555285Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:04:49.8557482Z 84b2c7a89982: Download complete 2024-06-26T07:05:38.0050257Z 3d914950f8ad: Pull complete 2024-06-26T07:05:38.2285678Z 96d0e2032184: Pull complete 2024-06-26T07:05:38.4853702Z 06e12ce59790: Pull complete 2024-06-26T07:05:38.7174060Z ed712f63cde7: Pull complete 2024-06-26T07:05:38.9416596Z a97a71ef8a4e: Pull complete 2024-06-26T07:05:39.1043547Z 25cd0c482ad6: Pull complete 2024-06-26T07:05:42.6516251Z 9bf94bc45658: Pull complete 2024-06-26T07:05:42.8660744Z de45a14c9b22: Pull complete 2024-06-26T07:05:43.0837994Z e99241e77209: Pull complete 2024-06-26T07:05:43.3061188Z 35e8f4e2a42f: Pull complete 2024-06-26T07:05:43.5237132Z a823ffa4c541: Pull complete 2024-06-26T07:05:43.7029135Z 44ba272eae20: Pull complete 2024-06-26T07:05:51.9901754Z da02d278e524: Pull complete 2024-06-26T07:05:52.2152969Z 73a0ce681c29: Pull complete 2024-06-26T07:05:52.4371188Z 49737bd5ae70: Pull complete 2024-06-26T07:05:53.4019830Z 9cd4c745f8e4: Pull complete 2024-06-26T07:05:53.6184146Z b2950a036534: Pull complete 2024-06-26T07:05:53.8530930Z 25e19d81d9fc: Pull complete 2024-06-26T07:05:54.3517112Z 60156b40fcbe: Pull complete 2024-06-26T07:05:54.5834803Z 7a868239227f: Pull complete 2024-06-26T07:05:55.0129594Z 83911d7c7ecf: Pull complete 2024-06-26T07:05:55.2308310Z b3c5d66c679f: Pull complete 2024-06-26T07:06:34.4328229Z 84b2c7a89982: Pull complete 2024-06-26T07:06:34.6573463Z c1068d03a811: Pull complete 2024-06-26T07:06:34.7429132Z 671b4a301d81: Pull complete 2024-06-26T07:06:36.7113773Z 67760cf5ec46: Pull complete 2024-06-26T07:06:36.7482672Z ddc9740bcea2: Pull complete 2024-06-26T07:06:36.7968758Z fb2e97768560: Pull complete 2024-06-26T07:06:36.9298848Z b599c5a8c7cb: Pull complete 2024-06-26T07:06:36.9725120Z e08e88c13668: Pull complete 2024-06-26T07:07:10.9499301Z 12ba7431ff38: Pull complete 2024-06-26T07:07:11.1764827Z cf7c2cb896cb: Pull complete 2024-06-26T07:07:11.2880835Z 3a778ddfac2a: Pull complete 2024-06-26T07:07:11.6865244Z 236c8f0245f4: Pull complete 2024-06-26T07:07:11.9012384Z 4f4fb700ef54: Pull complete 2024-06-26T07:07:12.0309097Z dda1ce302b9a: Pull complete 2024-06-26T07:07:12.1892945Z e797fe86caef: Pull complete 2024-06-26T07:07:12.5754545Z 967190076fc0: Pull complete 2024-06-26T07:07:12.7994623Z 4fb0a83f0685: Pull complete 2024-06-26T07:07:13.0198534Z 68af2ec4e30a: Pull complete 2024-06-26T07:07:13.2547588Z cc91933e985a: Pull complete 2024-06-26T07:07:13.4824403Z 9f3187916e7d: Pull complete 2024-06-26T07:07:14.5724431Z 526083994857: Pull complete 2024-06-26T07:07:14.7986172Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:07:16.8899104Z 6bc9dc8315b0: Pull complete 2024-06-26T07:07:17.1142626Z 1b0e27c35f07: Pull complete 2024-06-26T07:07:30.4072148Z 95237e7311a1: Pull complete 2024-06-26T07:07:30.4499376Z 789eff6c51aa: Pull complete 2024-06-26T07:07:30.4940769Z de0f8a8c45c6: Pull complete 2024-06-26T07:07:31.1902527Z 2060028f7488: Pull complete 2024-06-26T07:07:31.2331624Z e412c7f3b2de: Pull complete 2024-06-26T07:07:31.5139787Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:07:31.5170507Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:31.5197917Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:31.5249582Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:07:31.5250536Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:07:31.5258710Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:31.5259211Z env: 2024-06-26T07:07:31.5259488Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:31.5259842Z ##[endgroup] 2024-06-26T07:07:31.5417509Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:07:31.5418059Z with: 2024-06-26T07:07:31.5418350Z driver-version: 550.54.15 2024-06-26T07:07:31.5418693Z env: 2024-06-26T07:07:31.5418964Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:31.5419314Z ##[endgroup] 2024-06-26T07:07:31.5458668Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:07:31.5459236Z with: 2024-06-26T07:07:31.5459528Z timeout_minutes: 10 2024-06-26T07:07:31.5459863Z max_attempts: 3 2024-06-26T07:07:31.5490755Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:07:31.5523844Z retry_wait_seconds: 10 2024-06-26T07:07:31.5524219Z polling_interval_seconds: 1 2024-06-26T07:07:31.5524592Z warning_on_retry: true 2024-06-26T07:07:31.5524945Z continue_on_error: false 2024-06-26T07:07:31.5525287Z env: 2024-06-26T07:07:31.5525562Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:31.5525917Z DRIVER_VERSION: 550.54.15 2024-06-26T07:07:31.5526265Z ##[endgroup] 2024-06-26T07:07:31.6061086Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:07:31.6062188Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:07:31.6062837Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:07:31.9119211Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:31.9506727Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:07:31.9746479Z No Packages marked for removal 2024-06-26T07:07:31.9863560Z + install_nvidia_driver_common 2024-06-26T07:07:31.9865875Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:07:31.9866375Z + lspci 2024-06-26T07:07:31.9866672Z Before installing NVIDIA driver 2024-06-26T07:07:31.9944507Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:07:31.9945415Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:07:31.9946363Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:07:31.9947177Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:07:31.9948096Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:07:31.9949099Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:07:31.9949893Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:07:31.9950810Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:07:31.9951398Z + lsmod 2024-06-26T07:07:31.9960248Z Module Size Used by 2024-06-26T07:07:31.9960835Z nvidia_modeset 1339392 0 2024-06-26T07:07:31.9961327Z veth 16384 0 2024-06-26T07:07:31.9961775Z nvidia_uvm 4599808 0 2024-06-26T07:07:31.9962239Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:07:31.9962758Z drm 421888 1 nvidia 2024-06-26T07:07:31.9963377Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:07:31.9963854Z backlight 16384 1 nvidia_modeset 2024-06-26T07:07:31.9964348Z xt_conntrack 16384 1 2024-06-26T07:07:31.9964742Z ipt_MASQUERADE 16384 1 2024-06-26T07:07:31.9965202Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:07:31.9965664Z nf_conntrack_netlink 49152 0 2024-06-26T07:07:31.9966113Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:07:31.9966583Z xfrm_user 45056 1 2024-06-26T07:07:31.9966988Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:07:31.9967409Z iptable_nat 16384 1 2024-06-26T07:07:31.9967791Z nf_conntrack_ipv4 16384 3 2024-06-26T07:07:31.9968225Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:07:31.9968708Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:07:31.9969229Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:07:31.9970239Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:07:31.9971085Z xt_addrtype 16384 2 2024-06-26T07:07:31.9971473Z iptable_filter 16384 1 2024-06-26T07:07:31.9971844Z br_netfilter 24576 0 2024-06-26T07:07:31.9972261Z bridge 172032 1 br_netfilter 2024-06-26T07:07:31.9972707Z stp 16384 1 bridge 2024-06-26T07:07:31.9973133Z llc 16384 2 bridge,stp 2024-06-26T07:07:31.9973589Z overlay 86016 0 2024-06-26T07:07:31.9974079Z sunrpc 393216 1 2024-06-26T07:07:31.9974457Z dm_mirror 28672 0 2024-06-26T07:07:31.9974854Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:07:31.9975340Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:07:31.9975863Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:07:31.9976368Z dax 69632 1 dm_mod 2024-06-26T07:07:31.9976780Z crc32_pclmul 16384 0 2024-06-26T07:07:31.9977153Z ghash_clmulni_intel 16384 0 2024-06-26T07:07:31.9977533Z pcbc 16384 0 2024-06-26T07:07:31.9977900Z aesni_intel 188416 0 2024-06-26T07:07:31.9978308Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:07:31.9978941Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:07:31.9979579Z glue_helper 16384 1 aesni_intel 2024-06-26T07:07:31.9980340Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:07:31.9981099Z mousedev 24576 0 2024-06-26T07:07:31.9981650Z evdev 20480 3 2024-06-26T07:07:31.9982196Z psmouse 32768 0 2024-06-26T07:07:31.9982757Z button 16384 0 2024-06-26T07:07:31.9983262Z ena 139264 0 2024-06-26T07:07:31.9983775Z ptp 20480 1 ena 2024-06-26T07:07:31.9984334Z pps_core 20480 1 ptp 2024-06-26T07:07:31.9984933Z crc32c_intel 24576 0 2024-06-26T07:07:31.9985509Z autofs4 49152 2 2024-06-26T07:07:31.9985980Z + modinfo nvidia 2024-06-26T07:07:31.9986848Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:07:31.9988100Z alias: char-major-195-* 2024-06-26T07:07:31.9988639Z version: 550.54.15 2024-06-26T07:07:31.9989148Z supported: external 2024-06-26T07:07:31.9989642Z license: NVIDIA 2024-06-26T07:07:31.9990308Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:07:31.9991024Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:07:31.9991669Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:07:31.9992306Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:07:31.9992990Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:07:31.9993676Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:07:31.9994680Z depends: i2c-core,drm 2024-06-26T07:07:31.9995356Z retpoline: Y 2024-06-26T07:07:31.9995791Z name: nvidia 2024-06-26T07:07:31.9996558Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:07:31.9997452Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:07:31.9998297Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:07:31.9999080Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:07:31.9999692Z parm: NVreg_RmLogonRC:int 2024-06-26T07:07:32.0000286Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:07:32.0000918Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:07:32.0001526Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:07:32.0002135Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:07:32.0002852Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:07:32.0003611Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:07:32.0004282Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:07:32.0004973Z parm: NVreg_EnableMSI:int 2024-06-26T07:07:32.0005551Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:07:32.0006178Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:07:32.0006907Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:07:32.0007708Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:07:32.0008489Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:07:32.0009330Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:32.0010161Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:07:32.0011149Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:32.0011975Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:07:32.0012638Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:07:32.0013374Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:07:32.0014131Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:07:32.0014821Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:07:32.0015458Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:07:32.0016128Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:07:32.0016790Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:07:32.0017409Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:07:32.0018092Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:07:32.0018832Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:07:32.0019510Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:07:32.0020204Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:07:32.0020899Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:07:32.0021574Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:07:32.0022274Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:07:32.0022960Z parm: NVreg_RmMsg:charp 2024-06-26T07:07:32.0023542Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:07:32.0024210Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:07:32.0024898Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:07:32.0025552Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:07:32.0026241Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:07:32.0026930Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:07:32.0027589Z parm: rm_firmware_active:charp 2024-06-26T07:07:32.0028186Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:07:32.0028754Z ++ command -v nvidia-smi 2024-06-26T07:07:32.0029325Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:07:32.0029862Z + set +e 2024-06-26T07:07:32.0030694Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:07:33.9888771Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:07:33.9889405Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:07:33.9889901Z + '[' 0 -ne 0 ']' 2024-06-26T07:07:33.9890263Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:07:33.9890985Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:07:33.9891708Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:07:33.9892424Z + set -e 2024-06-26T07:07:33.9892726Z + '[' 1 -eq 0 ']' 2024-06-26T07:07:33.9893305Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:07:33.9893981Z + post_install_nvidia_driver_common 2024-06-26T07:07:33.9894402Z + sudo modprobe nvidia 2024-06-26T07:07:33.9990375Z + echo 'After installing NVIDIA driver' 2024-06-26T07:07:33.9990850Z + lspci 2024-06-26T07:07:33.9992728Z After installing NVIDIA driver 2024-06-26T07:07:34.0069841Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:07:34.0071011Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:07:34.0071938Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:07:34.0072746Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:07:34.0073491Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:07:34.0074236Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:07:34.0075220Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:07:34.0076003Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:07:34.0076580Z + lsmod 2024-06-26T07:07:34.0084232Z Module Size Used by 2024-06-26T07:07:34.0084855Z nvidia_modeset 1339392 0 2024-06-26T07:07:34.0085616Z veth 16384 0 2024-06-26T07:07:34.0086122Z nvidia_uvm 4599808 0 2024-06-26T07:07:34.0086721Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:07:34.0087293Z drm 421888 1 nvidia 2024-06-26T07:07:34.0087718Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:07:34.0088192Z backlight 16384 1 nvidia_modeset 2024-06-26T07:07:34.0088648Z xt_conntrack 16384 1 2024-06-26T07:07:34.0089040Z ipt_MASQUERADE 16384 1 2024-06-26T07:07:34.0089484Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:07:34.0089946Z nf_conntrack_netlink 49152 0 2024-06-26T07:07:34.0090399Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:07:34.0090864Z xfrm_user 45056 1 2024-06-26T07:07:34.0091266Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:07:34.0091682Z iptable_nat 16384 1 2024-06-26T07:07:34.0092068Z nf_conntrack_ipv4 16384 3 2024-06-26T07:07:34.0092513Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:07:34.0092994Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:07:34.0093694Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:07:34.0094958Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:07:34.0096100Z xt_addrtype 16384 2 2024-06-26T07:07:34.0096609Z iptable_filter 16384 1 2024-06-26T07:07:34.0097080Z br_netfilter 24576 0 2024-06-26T07:07:34.0097492Z bridge 172032 1 br_netfilter 2024-06-26T07:07:34.0097930Z stp 16384 1 bridge 2024-06-26T07:07:34.0098348Z llc 16384 2 bridge,stp 2024-06-26T07:07:34.0098766Z overlay 86016 0 2024-06-26T07:07:34.0099134Z sunrpc 393216 1 2024-06-26T07:07:34.0099503Z dm_mirror 28672 0 2024-06-26T07:07:34.0099888Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:07:34.0100384Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:07:34.0100892Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:07:34.0101347Z dax 69632 1 dm_mod 2024-06-26T07:07:34.0101788Z crc32_pclmul 16384 0 2024-06-26T07:07:34.0102165Z ghash_clmulni_intel 16384 0 2024-06-26T07:07:34.0102782Z pcbc 16384 0 2024-06-26T07:07:34.0103160Z aesni_intel 188416 0 2024-06-26T07:07:34.0103541Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:07:34.0103989Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:07:34.0104441Z glue_helper 16384 1 aesni_intel 2024-06-26T07:07:34.0104991Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:07:34.0105527Z mousedev 24576 0 2024-06-26T07:07:34.0105892Z evdev 20480 3 2024-06-26T07:07:34.0106256Z psmouse 32768 0 2024-06-26T07:07:34.0106624Z button 16384 0 2024-06-26T07:07:34.0106986Z ena 139264 0 2024-06-26T07:07:34.0107354Z ptp 20480 1 ena 2024-06-26T07:07:34.0107748Z pps_core 20480 1 ptp 2024-06-26T07:07:34.0108132Z crc32c_intel 24576 0 2024-06-26T07:07:34.0108497Z autofs4 49152 2 2024-06-26T07:07:34.0108851Z + modinfo nvidia 2024-06-26T07:07:34.0109513Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:07:34.0110192Z alias: char-major-195-* 2024-06-26T07:07:34.0110719Z version: 550.54.15 2024-06-26T07:07:34.0111072Z supported: external 2024-06-26T07:07:34.0111406Z license: NVIDIA 2024-06-26T07:07:34.0111776Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:07:34.0112256Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:07:34.0112711Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:07:34.0113173Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:07:34.0113745Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:07:34.0114240Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:07:34.0115043Z depends: i2c-core,drm 2024-06-26T07:07:34.0115406Z retpoline: Y 2024-06-26T07:07:34.0115710Z name: nvidia 2024-06-26T07:07:34.0116263Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:07:34.0116924Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:07:34.0117568Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:07:34.0118163Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:07:34.0118606Z parm: NVreg_RmLogonRC:int 2024-06-26T07:07:34.0119039Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:07:34.0119489Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:07:34.0119921Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:07:34.0120353Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:07:34.0120892Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:07:34.0121445Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:07:34.0121924Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:07:34.0122351Z parm: NVreg_EnableMSI:int 2024-06-26T07:07:34.0122763Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:07:34.0123220Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:07:34.0123749Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:07:34.0124325Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:07:34.0124871Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:07:34.0125473Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:34.0126063Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:07:34.0126671Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:34.0127248Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:07:34.0127751Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:07:34.0128280Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:07:34.0128819Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:07:34.0129303Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:07:34.0129874Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:07:34.0130367Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:07:34.0130841Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:07:34.0131279Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:07:34.0131780Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:07:34.0132305Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:07:34.0132779Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:07:34.0133256Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:07:34.0133749Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:07:34.0134241Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:07:34.0134738Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:07:34.0135253Z parm: NVreg_RmMsg:charp 2024-06-26T07:07:34.0135659Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:07:34.0136120Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:07:34.0136585Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:07:34.0137039Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:07:34.0137512Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:07:34.0137985Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:07:34.0138425Z parm: rm_firmware_active:charp 2024-06-26T07:07:34.0138824Z + set +e 2024-06-26T07:07:34.0139120Z + nvidia-smi 2024-06-26T07:07:35.5664105Z Wed Jun 26 07:07:35 2024 2024-06-26T07:07:35.5665045Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:35.5666007Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:07:35.5666974Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:07:35.5667764Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:07:35.5668647Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:07:35.5669337Z | | | MIG M. | 2024-06-26T07:07:35.5669886Z |=========================================+========================+======================| 2024-06-26T07:07:35.5806693Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:07:35.5807520Z | 0% 33C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:07:35.5808156Z | | | N/A | 2024-06-26T07:07:35.5809009Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:07:35.5809673Z 2024-06-26T07:07:35.5810319Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:35.5810958Z | Processes: | 2024-06-26T07:07:35.5811667Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:07:35.5812361Z | ID ID Usage | 2024-06-26T07:07:35.5812919Z |=========================================================================================| 2024-06-26T07:07:35.5813566Z | No running processes found | 2024-06-26T07:07:35.5814317Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:36.1794062Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:07:37.7365179Z NVIDIA A10G 2024-06-26T07:07:38.1738859Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:07:38.1739528Z + '[' 0 -eq 0 ']' 2024-06-26T07:07:38.1740020Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:07:38.1740877Z + set -e 2024-06-26T07:07:38.1741181Z INFO: Ignoring allowed status 0 2024-06-26T07:07:38.1742948Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:07:38.1744953Z + sudo yum install -y yum-utils 2024-06-26T07:07:38.4780143Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:39.9695572Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:07:39.9696275Z Nothing to do 2024-06-26T07:07:40.1188318Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:07:40.1189276Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:40.1190838Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:40.4346844Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:40.4594754Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:40.4595972Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:07:40.4596916Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:07:40.4707613Z + sudo yum install -y nvidia-docker2 2024-06-26T07:07:40.7758662Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:42.2537664Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:07:42.2538348Z Nothing to do 2024-06-26T07:07:42.4041760Z + sudo systemctl restart docker 2024-06-26T07:08:22.6533941Z Command completed after 1 attempt(s). 2024-06-26T07:08:22.6592297Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:08:22.6593074Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:08:22.6593762Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:08:22.6594588Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:22.6602300Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:22.6602805Z env: 2024-06-26T07:08:22.6603086Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:22.6603535Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:22.6604007Z ##[endgroup] 2024-06-26T07:08:22.8815087Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:08:22.8972262Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:08:22.9073570Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:08:23.0004816Z Prepare all required actions 2024-06-26T07:08:23.0005310Z Getting action download info 2024-06-26T07:08:23.0971173Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:08:23.2746048Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:08:24.5996177Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:08:24.5996656Z with: 2024-06-26T07:08:24.5996999Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:24.5997452Z s3-bucket: gha-artifacts 2024-06-26T07:08:24.5997797Z env: 2024-06-26T07:08:24.5998067Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:24.5998524Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:24.5999010Z ##[endgroup] 2024-06-26T07:08:24.6030655Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:08:24.6031114Z with: 2024-06-26T07:08:24.6031496Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:24.6031992Z s3-bucket: gha-artifacts 2024-06-26T07:08:24.6032337Z region: us-east-1 2024-06-26T07:08:24.6032636Z env: 2024-06-26T07:08:24.6032906Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:24.6033348Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:24.6033824Z ##[endgroup] 2024-06-26T07:08:25.0186182Z (node:9302) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:08:25.0186956Z 2024-06-26T07:08:25.0187219Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:08:25.0187955Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:08:25.0188965Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:08:25.1052328Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:08:25.1054536Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:08:32.5910745Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:08:32.5917625Z Artifact download has finished successfully 2024-06-26T07:08:32.6067802Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:08:32.6068245Z unzip -o artifacts.zip 2024-06-26T07:08:32.6077868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:32.6078375Z env: 2024-06-26T07:08:32.6078650Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:32.6079102Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:32.6079578Z ##[endgroup] 2024-06-26T07:08:32.6107986Z Archive: artifacts.zip 2024-06-26T07:08:32.6109572Z creating: dist/ 2024-06-26T07:08:34.5852056Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:08:34.5853674Z creating: build/custom_test_artifacts/ 2024-06-26T07:08:34.5854985Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:08:34.5855740Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:08:34.5856574Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:34.5858039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:34.5858971Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:34.5859911Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:34.5860910Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:34.5861889Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:34.5863003Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:34.5864237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:34.5865251Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:34.5866243Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:34.5867404Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:34.5868569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:34.5869705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:34.5870989Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:34.5872145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:34.5873319Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:34.5874596Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:34.5875611Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:34.5912422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:34.5951130Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:34.5953987Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:34.5994627Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:34.5996096Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:34.5997542Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:34.5999017Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:34.6000426Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:34.6001813Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:34.6003205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:34.6004582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:34.6006127Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:34.6007418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:34.6008657Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:34.6009869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:34.6011103Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:34.6012290Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:34.6013514Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:34.6066073Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:34.6123314Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:34.6124547Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:34.6125575Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:34.6126438Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:34.6127339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:34.6128263Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:08:34.6129293Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:08:34.6130445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:08:34.6131563Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:08:34.6132599Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:08:34.6133661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:08:34.6134743Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:08:34.6135823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:08:34.6136895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:08:34.6137951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:08:34.6150428Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:08:34.6278752Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:08:34.6279753Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:08:34.6280845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:08:34.6282047Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:08:34.6283213Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:08:34.6284306Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:08:34.6285482Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:08:34.6286618Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:08:34.6287866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:08:34.6289005Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:08:34.6290122Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:08:34.6304110Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:08:34.6384258Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:08:34.6385734Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:34.6386807Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:34.6387889Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:08:34.6388802Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:08:34.6389700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:34.6390756Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:08:34.6391604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:08:34.6392379Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:08:34.6393188Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:08:34.6498988Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:08:34.6559601Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:08:34.6560328Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:08:34.6561022Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:08:34.6561829Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:34.6567065Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:34.6568053Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:34.6568965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:34.6569940Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:34.6570882Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:34.6571966Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:34.6573042Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:34.6574026Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:34.6575006Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:34.6576131Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:34.6577263Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:34.6578380Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:34.6579490Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:34.6580626Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:34.6581764Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:34.6582901Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:34.6583883Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:34.6621645Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:34.6658611Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:34.6660039Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:34.6703814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:34.6705803Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:34.6707339Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:34.6708816Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:34.6710364Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:34.6711723Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:34.6713103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:34.6714698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:34.6716046Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:34.6717313Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:34.6718540Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:34.6719741Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:34.6720962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:34.6722138Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:34.6723349Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:34.6771881Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:34.6829688Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:34.6831168Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:34.6832240Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:34.6833097Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:34.6833978Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:34.6835148Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:08:34.6836231Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:08:34.6837422Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:08:34.6838689Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:08:34.6839760Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:08:34.6840872Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:08:34.6841985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:08:34.6843105Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:08:34.6844211Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:08:34.6845338Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:08:34.6856687Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:08:34.6918377Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:08:34.6919607Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:34.6920669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:34.6921638Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:08:34.6922536Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:08:34.6923427Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:34.6924318Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:08:34.6925199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:08:34.6925971Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:08:34.6926782Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:08:34.6971876Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:08:34.6972710Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:08:34.6973473Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:08:34.6974348Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:34.6979369Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:34.6980475Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:34.6981460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:34.6982500Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:34.6983526Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:34.6984676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:34.6985827Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:34.6986870Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:34.6987905Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:34.6989106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:34.6990431Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:34.6991614Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:34.6993034Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:34.6994252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:34.6995703Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:34.6996806Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:34.6997863Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:34.7031536Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:34.7069136Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:34.7070840Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:34.7113434Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:34.7115094Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:34.7136400Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:34.7137960Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:34.7139433Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:34.7140894Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:34.7142355Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:34.7143806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:34.7145223Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:34.7146579Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:34.7147886Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:34.7149169Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:34.7150591Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:34.7151846Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:34.7153112Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:34.7206879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:34.7265359Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:34.7268656Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:34.7271367Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:34.7273545Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:34.7276243Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:34.7277304Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:08:34.7278473Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:08:34.7279775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:08:34.7281010Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:08:34.7282176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:08:34.7283353Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:08:34.7284760Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:08:34.7286008Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:08:34.7287190Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:08:34.7288366Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:08:34.7289609Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:08:34.7398508Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:08:34.7399670Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:08:34.7400842Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:08:34.7402151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:08:34.7403388Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:08:34.7404578Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:08:34.7405800Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:08:34.7407030Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:08:34.7408262Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:08:34.7409487Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:08:34.7410711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:08:34.7424184Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:08:34.7478296Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:08:34.7479599Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:34.7480728Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:34.7481765Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:08:34.7482744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:08:34.7483703Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:34.7484846Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:08:34.7485756Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:08:34.7486595Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:08:34.7487456Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:08:34.7587246Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:08:34.7627757Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:08:34.7628448Z creating: build/lib/ 2024-06-26T07:08:34.7713476Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:08:34.8152093Z inflating: build/lib/libprotobuf.a 2024-06-26T07:08:34.8162176Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:08:34.8170195Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:08:34.8178013Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:08:34.8179107Z inflating: build/lib/libclog.a 2024-06-26T07:08:34.8197055Z inflating: build/lib/libnnpack.a 2024-06-26T07:08:34.8199376Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:08:34.8262265Z inflating: build/lib/libgtest.a 2024-06-26T07:08:34.8334532Z inflating: build/lib/libbenchmark.a 2024-06-26T07:08:34.8396602Z inflating: build/lib/libasmjit.a 2024-06-26T07:08:34.8404071Z inflating: build/lib/libittnotify.a 2024-06-26T07:08:34.8431332Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:08:34.8555956Z inflating: build/lib/libgloo.a 2024-06-26T07:08:34.8577179Z inflating: build/lib/libfmt.a 2024-06-26T07:08:34.8666449Z inflating: build/lib/libc10.so 2024-06-26T07:08:34.8668060Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:08:34.8669082Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:08:34.8670490Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:08:34.8689785Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:08:34.8707354Z inflating: build/lib/libgmock.a 2024-06-26T07:08:34.8708248Z inflating: build/lib/libgtest_main.a 2024-06-26T07:08:34.8708739Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:08:34.9194913Z inflating: build/lib/libprotoc.a 2024-06-26T07:08:34.9564034Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:08:35.9299303Z inflating: build/lib/libdnnl.a 2024-06-26T07:08:35.9848855Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:08:35.9901795Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:08:35.9902428Z inflating: build/lib/libgmock_main.a 2024-06-26T07:08:36.1109873Z inflating: build/lib/libfbgemm.a 2024-06-26T07:08:36.1354657Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:08:36.1858261Z inflating: build/lib/libkineto.a 2024-06-26T07:08:36.2037332Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:08:36.2078256Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:08:36.2748293Z inflating: build/lib/libonnx.a 2024-06-26T07:08:38.6051864Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:08:38.6055723Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:08:38.6059549Z inflating: build/lib/libshm.so 2024-06-26T07:08:40.5919808Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:08:40.5920615Z inflating: build/lib/libtorch.so 2024-06-26T07:08:40.5923654Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:08:41.3871857Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:08:41.5765871Z inflating: build/lib/libtorch_python.so 2024-06-26T07:08:41.5831532Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:08:41.5851494Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:08:41.5876048Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:08:41.5909210Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:08:41.5909936Z creating: build/bin/ 2024-06-26T07:08:41.5958195Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:08:41.6007910Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:08:41.6058022Z inflating: build/bin/c10_Device_test 2024-06-26T07:08:41.6115593Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:08:41.6166768Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:08:41.6214157Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:08:41.6263446Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:08:41.6316691Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:08:41.6370013Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:08:41.6424589Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:08:41.6493257Z inflating: build/bin/c10_cow_test 2024-06-26T07:08:41.6545296Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:08:41.6592603Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:08:41.6640677Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:08:41.6689656Z inflating: build/bin/c10_Half_test 2024-06-26T07:08:41.6743907Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:08:41.6797333Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:08:41.6845670Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:08:41.6901043Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:08:41.6951616Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:08:41.7000618Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:08:41.7047982Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:08:41.7098422Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:08:41.7151545Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:08:41.7200569Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:08:41.7254960Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:08:41.7308866Z inflating: build/bin/c10_complex_test 2024-06-26T07:08:41.7359954Z inflating: build/bin/c10_exception_test 2024-06-26T07:08:41.7408627Z inflating: build/bin/c10_flags_test 2024-06-26T07:08:41.7457162Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:08:41.7616024Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:08:41.7664854Z inflating: build/bin/c10_irange_test 2024-06-26T07:08:41.7717006Z inflating: build/bin/c10_lazy_test 2024-06-26T07:08:41.7772074Z inflating: build/bin/c10_logging_test 2024-06-26T07:08:41.7844471Z inflating: build/bin/c10_optional_test 2024-06-26T07:08:41.7905704Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:08:41.7957929Z inflating: build/bin/c10_registry_test 2024-06-26T07:08:41.8104811Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:08:41.8155870Z inflating: build/bin/c10_ssize_test 2024-06-26T07:08:41.8205503Z inflating: build/bin/c10_string_util_test 2024-06-26T07:08:41.8262724Z inflating: build/bin/c10_string_view_test 2024-06-26T07:08:41.8311323Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:08:41.8358430Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:08:41.8412724Z inflating: build/bin/c10_typeid_test 2024-06-26T07:08:41.8845126Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:08:41.9278982Z inflating: build/bin/protoc 2024-06-26T07:08:41.9330388Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:08:41.9382187Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:08:41.9432723Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:08:41.9484362Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:08:41.9535264Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:08:41.9586983Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:08:41.9634010Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:08:41.9685078Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:08:42.0002615Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:08:42.0334218Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:08:42.0681923Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:08:42.0732506Z inflating: build/bin/FileStoreTest 2024-06-26T07:08:42.0786809Z inflating: build/bin/TCPStoreTest 2024-06-26T07:08:42.0838113Z inflating: build/bin/HashStoreTest 2024-06-26T07:08:42.0851019Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:08:42.0903176Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:08:42.0906961Z inflating: build/bin/torch_shm_manager 2024-06-26T07:08:42.0909633Z inflating: build/bin/example_allreduce 2024-06-26T07:08:42.0963011Z inflating: build/bin/test_dist_autograd 2024-06-26T07:08:42.1029948Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:08:42.1032178Z inflating: build/bin/parallel_benchmark 2024-06-26T07:08:42.1097000Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:08:42.1105954Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:08:42.1438330Z inflating: build/bin/test_lazy 2024-06-26T07:08:42.2583430Z inflating: build/bin/test_api 2024-06-26T07:08:42.2654138Z inflating: build/bin/Dict_test 2024-06-26T07:08:42.2705662Z inflating: build/bin/Dimname_test 2024-06-26T07:08:42.2768247Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:08:42.2823916Z inflating: build/bin/NamedTensor_test 2024-06-26T07:08:42.2880766Z inflating: build/bin/apply_utils_test 2024-06-26T07:08:42.2937858Z inflating: build/bin/atest 2024-06-26T07:08:42.2998958Z inflating: build/bin/basic 2024-06-26T07:08:42.3051934Z inflating: build/bin/broadcast_test 2024-06-26T07:08:42.3101520Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:08:42.3157468Z inflating: build/bin/cpu_generator_test 2024-06-26T07:08:42.3208762Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:08:42.3299858Z inflating: build/bin/cpu_rng_test 2024-06-26T07:08:42.3348241Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:08:42.3396881Z inflating: build/bin/dlconvertor_test 2024-06-26T07:08:42.3453088Z inflating: build/bin/extension_backend_test 2024-06-26T07:08:42.3506801Z inflating: build/bin/half_test 2024-06-26T07:08:42.3599101Z inflating: build/bin/ivalue_test 2024-06-26T07:08:42.3647298Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:08:42.3700990Z inflating: build/bin/math_kernel_test 2024-06-26T07:08:42.3753420Z inflating: build/bin/memory_format_test 2024-06-26T07:08:42.3804756Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:08:42.3855975Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:08:42.3910135Z inflating: build/bin/native_test 2024-06-26T07:08:42.3959100Z inflating: build/bin/operator_name_test 2024-06-26T07:08:42.4008867Z inflating: build/bin/operators_test 2024-06-26T07:08:42.4059744Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:08:42.4124350Z inflating: build/bin/pow_test 2024-06-26T07:08:42.4180265Z inflating: build/bin/quantized_test 2024-06-26T07:08:42.4228662Z inflating: build/bin/reduce_ops_test 2024-06-26T07:08:42.4278435Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:08:42.4333039Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:08:42.4389821Z inflating: build/bin/scalar_test 2024-06-26T07:08:42.4440074Z inflating: build/bin/StorageUtils_test 2024-06-26T07:08:42.4490885Z inflating: build/bin/stride_properties_test 2024-06-26T07:08:42.4566856Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:08:42.4619416Z inflating: build/bin/test_parallel 2024-06-26T07:08:42.4622162Z inflating: build/bin/thread_init_test 2024-06-26T07:08:42.4676448Z inflating: build/bin/type_ptr_test 2024-06-26T07:08:42.4735897Z inflating: build/bin/type_test 2024-06-26T07:08:42.4786518Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:08:42.4787897Z inflating: build/bin/verify_api_visibility 2024-06-26T07:08:42.4854778Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:08:42.4904829Z inflating: build/bin/weakref_test 2024-06-26T07:08:42.4955028Z inflating: build/bin/wrapdim_test 2024-06-26T07:08:42.5004706Z inflating: build/bin/xla_tensor_test 2024-06-26T07:08:42.5063235Z inflating: build/bin/IListRef_test 2024-06-26T07:08:42.5165433Z inflating: build/bin/List_test 2024-06-26T07:08:42.5229697Z inflating: build/bin/KernelFunction_test 2024-06-26T07:08:42.5347828Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:08:42.5441247Z inflating: build/bin/kernel_function_test 2024-06-26T07:08:42.5565747Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:08:42.5665988Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:08:42.5726524Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:08:42.5820188Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:08:42.5868855Z inflating: build/bin/CppSignature_test 2024-06-26T07:08:42.5924109Z inflating: build/bin/backend_fallback_test 2024-06-26T07:08:42.5971012Z inflating: build/bin/op_allowlist_test 2024-06-26T07:08:42.6032573Z inflating: build/bin/inline_container_test 2024-06-26T07:08:42.6324663Z inflating: build/bin/op_registration_test 2024-06-26T07:08:42.6375947Z inflating: build/bin/cuda_apply_test 2024-06-26T07:08:42.6425080Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:08:42.6482479Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:08:42.6535224Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:08:42.6602988Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:08:42.6660644Z inflating: build/bin/cuda_complex_test 2024-06-26T07:08:42.6708133Z inflating: build/bin/cuda_device_test 2024-06-26T07:08:42.6763763Z inflating: build/bin/cuda_cub_test 2024-06-26T07:08:42.6812856Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:08:42.6875959Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:08:42.6931335Z inflating: build/bin/cuda_generator_test 2024-06-26T07:08:42.6979851Z inflating: build/bin/cuda_half_test 2024-06-26T07:08:42.7028984Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:08:42.7076799Z inflating: build/bin/cuda_optional_test 2024-06-26T07:08:42.7126667Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:08:42.7177392Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:08:42.7227039Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:08:42.7286342Z inflating: build/bin/cuda_stream_test 2024-06-26T07:08:42.7334062Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:08:42.7385176Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:08:42.7399040Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:08:42.7462385Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:08:42.7518398Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:08:42.7579528Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:08:42.7639288Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:08:42.8445644Z inflating: build/bin/test_tensorexpr 2024-06-26T07:08:42.9005156Z inflating: build/bin/test_jit 2024-06-26T07:08:42.9005932Z creating: .additional_ci_files/ 2024-06-26T07:08:42.9059117Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:08:42.9270348Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:08:42.9297460Z ##[group]Run rm artifacts.zip 2024-06-26T07:08:42.9297984Z rm artifacts.zip 2024-06-26T07:08:42.9305972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:42.9306607Z env: 2024-06-26T07:08:42.9306889Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:42.9307347Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:42.9307859Z ##[endgroup] 2024-06-26T07:08:42.9993550Z ##[group]Run df -H 2024-06-26T07:08:42.9993871Z df -H 2024-06-26T07:08:43.0002208Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:43.0002724Z env: 2024-06-26T07:08:43.0003000Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:43.0003459Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:43.0003942Z ##[endgroup] 2024-06-26T07:08:43.0029675Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:08:43.0030513Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:08:43.0031292Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:08:43.0031813Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:08:43.0032284Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:08:43.0032777Z /dev/nvme0n1p1 162G 43G 119G 27% / 2024-06-26T07:08:43.0070768Z Prepare all required actions 2024-06-26T07:08:43.0071217Z Getting action download info 2024-06-26T07:08:43.1189871Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:08:43.1190440Z with: 2024-06-26T07:08:43.1190694Z env: 2024-06-26T07:08:43.1190970Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:43.1191421Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:43.1191907Z ##[endgroup] 2024-06-26T07:08:43.1222754Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:08:43.1223198Z with: 2024-06-26T07:08:43.1223462Z name: td_results 2024-06-26T07:08:43.1223782Z s3-bucket: gha-artifacts 2024-06-26T07:08:43.1224217Z region: us-east-1 2024-06-26T07:08:43.1224556Z env: 2024-06-26T07:08:43.1224831Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:43.1225286Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:43.1225770Z ##[endgroup] 2024-06-26T07:08:43.5314752Z (node:9323) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:08:43.5315842Z 2024-06-26T07:08:43.5316255Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:08:43.5317402Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:08:43.5318761Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:08:43.6053834Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:08:43.6059552Z Artifact download has finished successfully 2024-06-26T07:08:43.6182909Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:08:43.6183401Z mkdir -p .additional_ci_files 2024-06-26T07:08:43.6183965Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:08:43.6191987Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:43.6192485Z env: 2024-06-26T07:08:43.6192763Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:43.6193255Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:43.6193735Z ##[endgroup] 2024-06-26T07:08:43.6228243Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:08:43.6241115Z ##[error]Process completed with exit code 1. 2024-06-26T07:08:43.6285694Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:08:43.6286178Z .github/scripts/parse_ref.py 2024-06-26T07:08:43.6293145Z shell: /usr/bin/bash -e {0} 2024-06-26T07:08:43.6293491Z env: 2024-06-26T07:08:43.6293767Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:43.6294222Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:43.6294706Z ##[endgroup] 2024-06-26T07:08:43.6526299Z Prepare all required actions 2024-06-26T07:08:43.6564074Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:08:43.6564530Z with: 2024-06-26T07:08:43.6565076Z github-token: *** 2024-06-26T07:08:43.6565387Z env: 2024-06-26T07:08:43.6565658Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:43.6566238Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:43.6566729Z ##[endgroup] 2024-06-26T07:08:43.6581368Z ##[group]Run set -eux 2024-06-26T07:08:43.6581711Z set -eux 2024-06-26T07:08:43.6582307Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:08:43.6589801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:43.6590411Z env: 2024-06-26T07:08:43.6590714Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:43.6591171Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:43.6591817Z GITHUB_TOKEN: *** 2024-06-26T07:08:43.6592122Z ##[endgroup] 2024-06-26T07:08:43.6611711Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0b34867d3f3dcf691 2024-06-26T07:08:44.8817895Z setting job-id=26688911606 2024-06-26T07:08:44.8819052Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:44.9013969Z Prepare all required actions 2024-06-26T07:08:44.9014423Z Getting action download info 2024-06-26T07:08:45.0172577Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:08:45.0173033Z with: 2024-06-26T07:08:45.0173483Z github-token: *** 2024-06-26T07:08:45.0181662Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:08:45.0190440Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:45.0191196Z env: 2024-06-26T07:08:45.0191478Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:45.0191929Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:45.0192410Z ##[endgroup] 2024-06-26T07:08:45.0232719Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:08:45.0233263Z with: 2024-06-26T07:08:45.0233535Z shell: bash 2024-06-26T07:08:45.0233839Z timeout_minutes: 10 2024-06-26T07:08:45.0234174Z max_attempts: 5 2024-06-26T07:08:45.0234733Z retry_wait_seconds: 30 2024-06-26T07:08:45.0235852Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:08:45.0237196Z polling_interval_seconds: 1 2024-06-26T07:08:45.0237577Z warning_on_retry: true 2024-06-26T07:08:45.0237923Z continue_on_error: false 2024-06-26T07:08:45.0238263Z env: 2024-06-26T07:08:45.0238541Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:45.0238993Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:45.0239626Z GITHUB_TOKEN: *** 2024-06-26T07:08:45.0239957Z ##[endgroup] 2024-06-26T07:08:45.0737568Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:08:45.2882285Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:08:45.3035786Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:08:45.3180353Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:08:45.3188600Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:08:45.3201378Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:08:45.3221144Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:08:45.3229089Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:08:46.0738373Z Command completed after 1 attempt(s). 2024-06-26T07:08:46.0779552Z ##[group]Run set -x 2024-06-26T07:08:46.0779886Z set -x 2024-06-26T07:08:46.0780182Z  2024-06-26T07:08:46.0780738Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:08:46.0781425Z # in runner workspace 2024-06-26T07:08:46.0781961Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:08:46.0789678Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:46.0790182Z env: 2024-06-26T07:08:46.0790640Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:46.0791095Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:46.0791571Z ##[endgroup] 2024-06-26T07:08:46.0813124Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:08:46.1013572Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:08:46.1014127Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:08:46.1014589Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:08:46.1014989Z  2024-06-26T07:08:46.1015554Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:08:46.1016244Z # in runner workspace 2024-06-26T07:08:46.1016821Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:08:46.1017473Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:08:46.1017939Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:08:46.1026338Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:08:46.1035443Z  --selected-test-configs "" \ 2024-06-26T07:08:46.1035907Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:08:46.1036324Z  --tag "${TAG}" \ 2024-06-26T07:08:46.1036718Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:08:46.1037162Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:08:46.1037595Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:08:46.1044831Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:46.1045333Z env: 2024-06-26T07:08:46.1045606Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:46.1046072Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:46.1046750Z GITHUB_TOKEN: *** 2024-06-26T07:08:46.1047435Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:46.1048173Z PR_NUMBER: 2024-06-26T07:08:46.1048482Z TAG: ciflow/inductor/129470 2024-06-26T07:08:46.1048841Z EVENT_NAME: push 2024-06-26T07:08:46.1049149Z SCHEDULE: 2024-06-26T07:08:46.1049423Z HEAD_BRANCH: 2024-06-26T07:08:46.1049710Z ##[endgroup] 2024-06-26T07:08:46.1084857Z Workflow: inductor 2024-06-26T07:08:46.1085776Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:46.3120605Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:08:46.4489690Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:08:46.4490152Z echo "Filtered matrix:" 2024-06-26T07:08:46.4498435Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:08:46.4507241Z  2024-06-26T07:08:46.4507522Z echo 2024-06-26T07:08:46.4507903Z echo "Is the current job unstable? False" 2024-06-26T07:08:46.4508358Z  2024-06-26T07:08:46.4508625Z echo 2024-06-26T07:08:46.4508983Z echo "Is keep-going label set? False" 2024-06-26T07:08:46.4509427Z  2024-06-26T07:08:46.4509735Z echo 2024-06-26T07:08:46.4510048Z echo "Renabled issues? " 2024-06-26T07:08:46.4517955Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:46.4518458Z env: 2024-06-26T07:08:46.4518728Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:46.4519205Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:46.4519721Z ##[endgroup] 2024-06-26T07:08:46.4538993Z Filtered matrix: 2024-06-26T07:08:46.4548792Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:08:46.4557288Z 2024-06-26T07:08:46.4557446Z Is the current job unstable? False 2024-06-26T07:08:46.4557727Z 2024-06-26T07:08:46.4557968Z Is keep-going label set? False 2024-06-26T07:08:46.4558230Z 2024-06-26T07:08:46.4558355Z Renabled issues? 2024-06-26T07:08:46.4599821Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:46.4600526Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:46.4608326Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:46.4608813Z env: 2024-06-26T07:08:46.4609090Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:46.4609542Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:46.4610029Z JOB_TIMEOUT: 240 2024-06-26T07:08:46.4610331Z ##[endgroup] 2024-06-26T07:08:46.4679436Z ##[group]Run set -x 2024-06-26T07:08:46.4679826Z set -x 2024-06-26T07:08:46.4680122Z  2024-06-26T07:08:46.4680472Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:08:46.4681008Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:08:46.4681567Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:08:46.4682077Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:08:46.4682487Z else 2024-06-26T07:08:46.4682824Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:08:46.4683243Z fi 2024-06-26T07:08:46.4683518Z  2024-06-26T07:08:46.4683989Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:08:46.4684733Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:08:46.4685408Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:08:46.4685992Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:08:46.4686452Z container_name=$(docker run \ 2024-06-26T07:08:46.4686869Z  ${GPU_FLAG:-} \ 2024-06-26T07:08:46.4687244Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:08:46.4687642Z  -e PR_NUMBER \ 2024-06-26T07:08:46.4687997Z  -e GITHUB_ACTIONS \ 2024-06-26T07:08:46.4688389Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:08:46.4688795Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:08:46.4689207Z  -e GITHUB_JOB \ 2024-06-26T07:08:46.4689587Z  -e GITHUB_RUN_ID \ 2024-06-26T07:08:46.4689973Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:08:46.4690383Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:08:46.4690770Z  -e JOB_ID \ 2024-06-26T07:08:46.4691112Z  -e JOB_NAME \ 2024-06-26T07:08:46.4691462Z  -e BASE_SHA \ 2024-06-26T07:08:46.4691806Z  -e BRANCH \ 2024-06-26T07:08:46.4692130Z  -e SHA1 \ 2024-06-26T07:08:46.4692470Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:08:46.4692872Z  -e IN_WHEEL_TEST \ 2024-06-26T07:08:46.4693249Z  -e SHARD_NUMBER \ 2024-06-26T07:08:46.4693613Z  -e TEST_CONFIG \ 2024-06-26T07:08:46.4693979Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:08:46.4694368Z  -e REENABLED_ISSUES \ 2024-06-26T07:08:46.4694778Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:08:46.4695197Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:08:46.4695594Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:08:46.4695968Z  -e NO_TD \ 2024-06-26T07:08:46.4696307Z  -e TD_DISTRIBUTED \ 2024-06-26T07:08:46.4696684Z  -e PR_LABELS \ 2024-06-26T07:08:46.4697094Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:08:46.4697548Z  -e SCCACHE_BUCKET \ 2024-06-26T07:08:46.4697934Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:08:46.4698338Z  -e XLA_CUDA \ 2024-06-26T07:08:46.4698741Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:08:46.4699296Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:08:46.4699824Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:08:46.4700332Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:08:46.4700796Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:08:46.4701212Z  -e DASHBOARD_TAG \ 2024-06-26T07:08:46.4701682Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:08:46.4702236Z  --security-opt seccomp=unconfined \ 2024-06-26T07:08:46.4702699Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:08:46.4703087Z  --ipc=host \ 2024-06-26T07:08:46.4703536Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:08:46.4703932Z  --tty \ 2024-06-26T07:08:46.4704247Z  --detach \ 2024-06-26T07:08:46.4704604Z  --name="${container_name}" \ 2024-06-26T07:08:46.4705012Z  --user jenkins \ 2024-06-26T07:08:46.4705549Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:08:46.4706111Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:08:46.4706544Z  "${DOCKER_IMAGE}" 2024-06-26T07:08:46.4706885Z ) 2024-06-26T07:08:46.4707285Z # Propagate download.pytorch.org IP to container 2024-06-26T07:08:46.4708206Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:08:46.4709163Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:08:46.4710063Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:08:46.4717637Z shell: /usr/bin/bash -e {0} 2024-06-26T07:08:46.4717990Z env: 2024-06-26T07:08:46.4718270Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:46.4718724Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:46.4719380Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:46.4719885Z PR_NUMBER: 2024-06-26T07:08:46.4720210Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:08:46.4720629Z GITHUB_WORKFLOW: inductor 2024-06-26T07:08:46.4720975Z GITHUB_JOB: test 2024-06-26T07:08:46.4721296Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:08:46.4721660Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:08:46.4722007Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:08:46.4722341Z JOB_ID: 26688911606 2024-06-26T07:08:46.4723033Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:46.4723784Z BRANCH: 2024-06-26T07:08:46.4724114Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:46.4724637Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:46.4725144Z TEST_CONFIG: dynamic_inductor_huggingface 2024-06-26T07:08:46.4725578Z SHARD_NUMBER: 1 2024-06-26T07:08:46.4725884Z NUM_TEST_SHARDS: 1 2024-06-26T07:08:46.4726211Z REENABLED_ISSUES: 2024-06-26T07:08:46.4726558Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:08:46.4726946Z VERBOSE_TEST_LOGS: False 2024-06-26T07:08:46.4727294Z NO_TEST_TIMEOUT: False 2024-06-26T07:08:46.4727636Z NO_TD: False 2024-06-26T07:08:46.4727940Z TD_DISTRIBUTED: False 2024-06-26T07:08:46.4728361Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:08:46.4728852Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:08:46.4729285Z SHM_SIZE: 2g 2024-06-26T07:08:46.4730364Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:46.4731525Z XLA_CUDA: 2024-06-26T07:08:46.4732001Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:08:46.4732619Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:08:46.4733065Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:08:46.4733477Z DASHBOARD_TAG: 2024-06-26T07:08:46.4733982Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:08:46.4734363Z ##[endgroup] 2024-06-26T07:08:46.4756528Z + [[ dynamic_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:08:46.4757953Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:08:46.4758910Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:08:46.4763519Z +++ nproc --ignore=2 2024-06-26T07:08:46.4776237Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:57.6841921Z + container_name=1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T07:08:57.6844453Z + docker exec -i 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:08:57.6845338Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:08:57.7560440Z + echo DOCKER_CONTAINER_ID=1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T07:08:57.7562197Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:08:57.7564214Z + docker exec -t 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:08:58.1390724Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:08:58.4502114Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:08:58.4506949Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:08:58.4509233Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:08:58.4512594Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:08:58.4516021Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:08:58.4518237Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:08:58.4544127Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:08:58.4603082Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:08:58.5021588Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:08:58.5177243Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:08:59.7909750Z Installing collected packages: torch 2024-06-26T07:09:08.4411135Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:09:08.4412546Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:09:08.4413470Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:09:08.5153497Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:09:08.5160347Z + source .ci/pytorch/common.sh 2024-06-26T07:09:08.5162517Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:09:08.5167806Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:09:08.5169455Z +++ declare -f -t trap_add 2024-06-26T07:09:08.5175363Z ++ set -ex 2024-06-26T07:09:08.5176035Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:09:08.5176697Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:09:08.5177210Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:09:08.5179460Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:09:08.5187509Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:09:08.5187949Z + trap_add cleanup_workspace EXIT 2024-06-26T07:09:08.5188357Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:09:08.5188726Z + shift 2024-06-26T07:09:08.5189012Z + for trap_add_name in "$@" 2024-06-26T07:09:08.5192705Z +++ trap -p EXIT 2024-06-26T07:09:08.5194157Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:09:08.5194873Z +++ extract_trap_cmd 2024-06-26T07:09:08.5195258Z +++ printf '%s\n' '' 2024-06-26T07:09:08.5195663Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:09:08.5196834Z + trap -- ' 2024-06-26T07:09:08.5197313Z cleanup_workspace' EXIT 2024-06-26T07:09:08.5197831Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:09:09.0284593Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:09:09.0295163Z + echo 'Environment variables:' 2024-06-26T07:09:09.0295631Z Environment variables: 2024-06-26T07:09:09.0295946Z + env 2024-06-26T07:09:09.0301021Z INSTALLED_DB=yes 2024-06-26T07:09:09.0301574Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:09:09.0301963Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:09:09.0302335Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:09:09.0302919Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:09:09.0303557Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:09:09.0304146Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:09:09.0304776Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:09.0305416Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:09.0305920Z HOSTNAME=1d5ca5302dc6 2024-06-26T07:09:09.0306861Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0307944Z GITHUB_ACTION=__self 2024-06-26T07:09:09.0308299Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:09:09.0313179Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:09:09.0317925Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:09:09.0318438Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:09:09.0318788Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:09:09.0319159Z TEST_CONFIG=dynamic_inductor_huggingface 2024-06-26T07:09:09.0319591Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:09:09.0320128Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:09:09.0320588Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:09:09.0321078Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:09:09.0321493Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:09:09.0322194Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:09.0322689Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:09:09.0323146Z GITHUB_REF_TYPE=tag 2024-06-26T07:09:09.0323469Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:09:09.0323849Z NCCL_VERSION=2.17.1-1 2024-06-26T07:09:09.0324231Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0324766Z XLA_CUDA= 2024-06-26T07:09:09.0325275Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:09:09.0325961Z *** 2024-06-26T07:09:09.0326249Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:09:09.0326635Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:09:09.0326998Z GITHUB_ACTIONS=true 2024-06-26T07:09:09.0327326Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:09.0327817Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:09:09.0328343Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:09:09.0328808Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0329302Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:09:09.0329791Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0330569Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:09:09.0331286Z UCC_HOME=/usr 2024-06-26T07:09:09.0331615Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:09:09.0332017Z VERBOSE_TEST_LOGS=False 2024-06-26T07:09:09.0332360Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:09:09.0332811Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:09:09.0333310Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:09:09.0333767Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:09:09.0334141Z SHARD_NUMBER=1 2024-06-26T07:09:09.0334450Z GITHUB_REF_PROTECTED=false 2024-06-26T07:09:09.0334801Z HOME=/var/lib/jenkins 2024-06-26T07:09:09.0335165Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:09:09.0335611Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:09:09.0336091Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:09:09.0336575Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:09:09.0336939Z CUDA_VERSION=12.1.1 2024-06-26T07:09:09.0337359Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:09:09.0337798Z NUM_TEST_SHARDS=1 2024-06-26T07:09:09.0338091Z UCX_HOME=/usr 2024-06-26T07:09:09.0338606Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:09:09.0339682Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0340959Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:09.0342205Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0343309Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:09:09.0343969Z GITHUB_EVENT_NAME=push 2024-06-26T07:09:09.0344296Z DASHBOARD_TAG= 2024-06-26T07:09:09.0344597Z GITHUB_RUN_ID=9673646808 2024-06-26T07:09:09.0345059Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:09:09.0345590Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:09:09.0346574Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0347476Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:09.0347873Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:09:09.0348252Z PR_NUMBER= 2024-06-26T07:09:09.0348534Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:09:09.0348866Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:09:09.0349311Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:09:09.0349770Z TERM=xterm 2024-06-26T07:09:09.0350202Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:09:09.0350608Z INSTALLED_VISION=yes 2024-06-26T07:09:09.0350923Z BRANCH= 2024-06-26T07:09:09.0351200Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:09:09.0351590Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:09:09.0352102Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:09:09.0352859Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:09:09.0353602Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:09:09.0354082Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:09:09.0354788Z REENABLED_ISSUES= 2024-06-26T07:09:09.0355284Z SHLVL=1 2024-06-26T07:09:09.0355555Z MAX_JOBS=14 2024-06-26T07:09:09.0355865Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:09:09.0356215Z NVARCH=x86_64 2024-06-26T07:09:09.0356532Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:09:09.0356994Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0357525Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:09:09.0358000Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:09:09.0358652Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:09:09.0359207Z GITHUB_JOB=test 2024-06-26T07:09:09.0359593Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:09:09.0360153Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:09:09.0360656Z NO_TEST_TIMEOUT=False 2024-06-26T07:09:09.0360976Z TD_DISTRIBUTED=False 2024-06-26T07:09:09.0361344Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:09:09.0361776Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:09:09.0362197Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:09:09.0362550Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:09:09.0362892Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:09:09.0363235Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:09:09.0364255Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:09.0365311Z GITHUB_BASE_REF= 2024-06-26T07:09:09.0365636Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:09:09.0366003Z CI=true 2024-06-26T07:09:09.0366313Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:09.0366725Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:09:09.0367150Z JOB_ID=26688911606 2024-06-26T07:09:09.0367465Z INSTALLED_PROTOBUF=yes 2024-06-26T07:09:09.0367789Z GITHUB_HEAD_REF= 2024-06-26T07:09:09.0368090Z GITHUB_ACTION_REF= 2024-06-26T07:09:09.0368525Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:09:09.0368992Z GITHUB_WORKFLOW=inductor 2024-06-26T07:09:09.0369359Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:09:09.0370270Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0371080Z NO_TD=False 2024-06-26T07:09:09.0371374Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:09:09.0371745Z _=/usr/bin/env 2024-06-26T07:09:09.0372224Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:09:09.0446199Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:09:09.0447395Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:09:09.0448505Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:09:09.0449473Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:09:09.0450133Z + BUILD_DIR=build 2024-06-26T07:09:09.0450457Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:09:09.0450895Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:09:09.0451242Z + SHARD_NUMBER=1 2024-06-26T07:09:09.0451540Z + NUM_TEST_SHARDS=1 2024-06-26T07:09:09.0451843Z + export VALGRIND=ON 2024-06-26T07:09:09.0452154Z + VALGRIND=ON 2024-06-26T07:09:09.0452617Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:09:09.0453103Z + [[ 0 == \1 ]] 2024-06-26T07:09:09.0453385Z + [[ False == \1 ]] 2024-06-26T07:09:09.0453843Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:09:09.0454365Z ++ realpath build/custom_test_artifacts 2024-06-26T07:09:09.0457174Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:09:09.0458222Z + [[ -n '' ]] 2024-06-26T07:09:09.0458568Z + echo 'Environment variables' 2024-06-26T07:09:09.0458944Z Environment variables 2024-06-26T07:09:09.0459263Z + env 2024-06-26T07:09:09.0462786Z INSTALLED_DB=yes 2024-06-26T07:09:09.0463355Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:09:09.0463876Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:09:09.0464536Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:09:09.0465281Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:09:09.0466094Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:09:09.0466754Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:09:09.0467421Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:09.0468065Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:09.0468649Z HOSTNAME=1d5ca5302dc6 2024-06-26T07:09:09.0469745Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0470838Z GITHUB_ACTION=__self 2024-06-26T07:09:09.0471234Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:09:09.0476118Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:09:09.0479986Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:09:09.0480493Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:09:09.0480861Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:09:09.0481232Z TEST_CONFIG=dynamic_inductor_huggingface 2024-06-26T07:09:09.0481665Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:09:09.0482128Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:09:09.0482586Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:09:09.0483023Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:09:09.0483430Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:09:09.0483863Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:09.0484352Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:09:09.0484817Z GITHUB_REF_TYPE=tag 2024-06-26T07:09:09.0485136Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:09:09.0485508Z NCCL_VERSION=2.17.1-1 2024-06-26T07:09:09.0485891Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0486323Z XLA_CUDA= 2024-06-26T07:09:09.0486774Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:09:09.0487158Z *** 2024-06-26T07:09:09.0487435Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:09:09.0487832Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:09:09.0488188Z GITHUB_ACTIONS=true 2024-06-26T07:09:09.0488513Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:09.0488993Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:09:09.0489512Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:09:09.0489974Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0490467Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:09:09.0490943Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0491717Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:09:09.0492418Z UCC_HOME=/usr 2024-06-26T07:09:09.0492747Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:09:09.0493140Z VERBOSE_TEST_LOGS=False 2024-06-26T07:09:09.0493472Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:09:09.0493918Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:09:09.0494407Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:09:09.0495003Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:09:09.0495373Z SHARD_NUMBER=1 2024-06-26T07:09:09.0495678Z GITHUB_REF_PROTECTED=false 2024-06-26T07:09:09.0496027Z HOME=/var/lib/jenkins 2024-06-26T07:09:09.0496380Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:09:09.0496867Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:09:09.0497417Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:09:09.0497900Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:09:09.0498257Z CUDA_VERSION=12.1.1 2024-06-26T07:09:09.0498671Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:09:09.0499106Z NUM_TEST_SHARDS=1 2024-06-26T07:09:09.0499402Z UCX_HOME=/usr 2024-06-26T07:09:09.0499900Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:09:09.0500963Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0502219Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:09.0503462Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0504555Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:09:09.0505211Z GITHUB_EVENT_NAME=push 2024-06-26T07:09:09.0505539Z DASHBOARD_TAG= 2024-06-26T07:09:09.0505839Z GITHUB_RUN_ID=9673646808 2024-06-26T07:09:09.0506276Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:09:09.0506808Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:09:09.0507787Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0508685Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:09.0509077Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:09:09.0509446Z PR_NUMBER= 2024-06-26T07:09:09.0509725Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:09:09.0510162Z VALGRIND=ON 2024-06-26T07:09:09.0510460Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:09:09.0510895Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:09:09.0511350Z TERM=xterm 2024-06-26T07:09:09.0511669Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:09:09.0512078Z INSTALLED_VISION=yes 2024-06-26T07:09:09.0512385Z BRANCH= 2024-06-26T07:09:09.0512675Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:09:09.0513069Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:09:09.0513484Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:09:09.0514235Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:09:09.0515079Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:09:09.0515566Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:09:09.0516030Z REENABLED_ISSUES= 2024-06-26T07:09:09.0516322Z SHLVL=1 2024-06-26T07:09:09.0516596Z MAX_JOBS=14 2024-06-26T07:09:09.0516929Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:09:09.0517282Z NVARCH=x86_64 2024-06-26T07:09:09.0517573Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:09:09.0518017Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:09.0518546Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:09:09.0519025Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:09:09.0519678Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:09:09.0520250Z GITHUB_JOB=test 2024-06-26T07:09:09.0520643Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:09:09.0521203Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:09:09.0521707Z NO_TEST_TIMEOUT=False 2024-06-26T07:09:09.0522032Z TD_DISTRIBUTED=False 2024-06-26T07:09:09.0522409Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:09:09.0522848Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:09:09.0523256Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:09:09.0535896Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:09:09.0536267Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:09:09.0536747Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:09:09.0537844Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:09.0538905Z GITHUB_BASE_REF= 2024-06-26T07:09:09.0539236Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:09:09.0539672Z CI=true 2024-06-26T07:09:09.0540012Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:09.0540431Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:09:09.0540798Z JOB_ID=26688911606 2024-06-26T07:09:09.0541115Z INSTALLED_PROTOBUF=yes 2024-06-26T07:09:09.0541443Z GITHUB_HEAD_REF= 2024-06-26T07:09:09.0541748Z GITHUB_ACTION_REF= 2024-06-26T07:09:09.0542173Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:09:09.0542644Z GITHUB_WORKFLOW=inductor 2024-06-26T07:09:09.0543003Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:09:09.0543901Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_2055c17a-1407-47b5-9d52-1555574d4e72 2024-06-26T07:09:09.0544717Z NO_TD=False 2024-06-26T07:09:09.0545026Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:09:09.0545391Z _=/usr/bin/env 2024-06-26T07:09:09.0545743Z + echo 'Testing pytorch' 2024-06-26T07:09:09.0546083Z Testing pytorch 2024-06-26T07:09:09.0546405Z + export LANG=C.UTF-8 2024-06-26T07:09:09.0546745Z + LANG=C.UTF-8 2024-06-26T07:09:09.0547089Z + PR_NUMBER= 2024-06-26T07:09:09.0547470Z + [[ dynamic_inductor_huggingface == \d\e\f\a\u\l\t ]] 2024-06-26T07:09:09.0548068Z + [[ dynamic_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:09:09.0548634Z + [[ dynamic_inductor_huggingface == \s\l\o\w ]] 2024-06-26T07:09:09.0549291Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:09:09.0550071Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:09:09.0550622Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:09:09.0551097Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:09:09.0551569Z + [[ dynamic_inductor_huggingface == *crossref* ]] 2024-06-26T07:09:09.0552177Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:09:09.0552807Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:09:09.0553452Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:09.0553996Z + pip_install --user ninja==1.10.2 2024-06-26T07:09:09.0554655Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:09:09.4081383Z Collecting ninja==1.10.2 2024-06-26T07:09:09.4249173Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:09:09.4387757Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:09:10.7053552Z Installing collected packages: ninja 2024-06-26T07:09:10.7118898Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:09:10.7120151Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:09:10.7157204Z Successfully installed ninja-1.10.2 2024-06-26T07:09:10.7987625Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:10.7990019Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:10.7991447Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:09:10.7991950Z + install_tlparse 2024-06-26T07:09:10.7992318Z + pip_install --user tlparse==0.3.7 2024-06-26T07:09:10.7992860Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:09:11.1466003Z Collecting tlparse==0.3.7 2024-06-26T07:09:11.1619728Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:09:11.1720657Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:09:12.4791160Z Installing collected packages: tlparse 2024-06-26T07:09:12.5155021Z Successfully installed tlparse-0.3.7 2024-06-26T07:09:12.5991970Z ++ python -m site --user-base 2024-06-26T07:09:12.6143594Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:12.6145434Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:09:12.6146280Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:09:12.6146944Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:12.6147877Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:09:12.6149030Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:09:12.6149726Z + cd test 2024-06-26T07:09:12.6150395Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:09:14.0792030Z + [[ dynamic_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:09:14.0792868Z + [[ dynamic_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:09:14.0793683Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:14.0794265Z + pushd test 2024-06-26T07:09:14.0794875Z ~/workspace/test ~/workspace 2024-06-26T07:09:14.0795408Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:09:15.5269878Z + CUDA_VERSION=12.1 2024-06-26T07:09:15.5270531Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:09:15.5270996Z + ISCUDA124= 2024-06-26T07:09:15.5271304Z + popd 2024-06-26T07:09:15.5271576Z ~/workspace 2024-06-26T07:09:15.5276102Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:09:15.5276793Z + [[ dynamic_inductor_huggingface == *dynamo_eager* ]] 2024-06-26T07:09:15.5277575Z + [[ dynamic_inductor_huggingface == *aot_eager* ]] 2024-06-26T07:09:15.5278123Z + [[ dynamic_inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:09:15.5278665Z + [[ dynamic_inductor_huggingface == *inductor* ]] 2024-06-26T07:09:15.5279204Z + [[ dynamic_inductor_huggingface != *perf* ]] 2024-06-26T07:09:15.5279742Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:09:15.5280204Z + [[ dynamic_inductor_huggingface == *dynamic* ]] 2024-06-26T07:09:15.5280848Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-26T07:09:15.5281468Z + [[ dynamic_inductor_huggingface == *cpu_inductor* ]] 2024-06-26T07:09:15.5282014Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:09:15.5304069Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:09:15.5304911Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:09:15.5306281Z + cd test 2024-06-26T07:09:15.5307321Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:09:16.8834631Z PyTorch built with: 2024-06-26T07:09:16.8835254Z - GCC 9.4 2024-06-26T07:09:16.8835583Z - C++ Version: 201703 2024-06-26T07:09:16.8836440Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:09:16.8837540Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:09:16.8838197Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:09:16.8838698Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:09:16.8839185Z - NNPACK is enabled 2024-06-26T07:09:16.8839568Z - CPU capability usage: AVX2 2024-06-26T07:09:16.8840020Z - CUDA Runtime 12.1 2024-06-26T07:09:16.8840527Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:09:16.8841123Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:09:16.8841561Z - Magma 2.6.1 2024-06-26T07:09:16.8848742Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:09:16.8855581Z 2024-06-26T07:09:17.1630539Z + cd test 2024-06-26T07:09:17.1631236Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:09:18.3657242Z ATen/Parallel: 2024-06-26T07:09:18.3657780Z at::get_num_threads() : 8 2024-06-26T07:09:18.3658305Z at::get_num_interop_threads() : 16 2024-06-26T07:09:18.3658735Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:09:18.3659142Z omp_get_max_threads() : 8 2024-06-26T07:09:18.3660106Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:09:18.3660903Z mkl_get_max_threads() : 8 2024-06-26T07:09:18.3661522Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:09:18.3662160Z std::thread::hardware_concurrency() : 16 2024-06-26T07:09:18.3662618Z Environment variables: 2024-06-26T07:09:18.3662962Z OMP_NUM_THREADS : [not set] 2024-06-26T07:09:18.3663339Z MKL_NUM_THREADS : [not set] 2024-06-26T07:09:18.3663718Z ATen parallel backend: OpenMP 2024-06-26T07:09:18.3663971Z 2024-06-26T07:09:18.6067842Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:09:18.6068592Z + [[ dynamic_inductor_huggingface == *backward* ]] 2024-06-26T07:09:18.6069110Z + [[ dynamic_inductor_huggingface == *xla* ]] 2024-06-26T07:09:18.6069628Z + [[ dynamic_inductor_huggingface == *executorch* ]] 2024-06-26T07:09:18.6070300Z + [[ dynamic_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:09:18.6071139Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:09:18.6071763Z + [[ dynamic_inductor_huggingface == distributed ]] 2024-06-26T07:09:18.6072334Z + [[ dynamic_inductor_huggingface == *inductor_distributed* ]] 2024-06-26T07:09:18.6072989Z + [[ dynamic_inductor_huggingface == *inductor-halide* ]] 2024-06-26T07:09:18.6073688Z + [[ dynamic_inductor_huggingface == *inductor-micro-benchmark* ]] 2024-06-26T07:09:18.6074300Z + [[ dynamic_inductor_huggingface == *huggingface* ]] 2024-06-26T07:09:18.6075020Z + install_torchvision 2024-06-26T07:09:18.6075351Z + local orig_preload 2024-06-26T07:09:18.6075667Z + local commit 2024-06-26T07:09:18.6075982Z ++ get_pinned_commit vision 2024-06-26T07:09:18.6076376Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:09:18.6081326Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:18.6081897Z + orig_preload= 2024-06-26T07:09:18.6082244Z + '[' -n '' ']' 2024-06-26T07:09:18.6083084Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:18.6084498Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:18.9163727Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:18.9166401Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-btf9n4vi 2024-06-26T07:09:18.9182121Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-btf9n4vi 2024-06-26T07:09:20.2799296Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:09:20.2817120Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:21.4915937Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:21.7155703Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:23.9941287Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:09:23.9990254Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:09:23.9993944Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:09:23.9999384Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:09:24.0200091Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:09:24.0206969Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:09:24.0210031Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:09:24.0213333Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:09:24.0216593Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:09:24.0219334Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:09:24.0649115Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:09:24.0801277Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:09:24.0882790Z Building wheels for collected packages: torchvision 2024-06-26T07:10:36.4460443Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:10:36.4487257Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112759 sha256=2babb4f16f6560cdaf3ab80867af98d89e163f96c08e5674a5d95bdf72c3666d 2024-06-26T07:10:36.4488971Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:10:36.4524449Z Successfully built torchvision 2024-06-26T07:10:37.6450429Z Installing collected packages: torchvision 2024-06-26T07:10:38.0325531Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:10:38.1402275Z + '[' -n '' ']' 2024-06-26T07:10:38.1404244Z + id=0 2024-06-26T07:10:38.1405071Z + test_dynamo_benchmark huggingface 0 2024-06-26T07:10:38.1405953Z ++ pwd 2024-06-26T07:10:38.1407750Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:38.1408737Z + local suite=huggingface 2024-06-26T07:10:38.1409082Z + shift 2024-06-26T07:10:38.1409350Z + local shard_id=0 2024-06-26T07:10:38.1409640Z + shift 2024-06-26T07:10:38.1410044Z + [[ dynamic_inductor_huggingface == *perf_compare* ]] 2024-06-26T07:10:38.1410901Z + [[ dynamic_inductor_huggingface == *perf* ]] 2024-06-26T07:10:38.1411501Z + [[ dynamic_inductor_huggingface == *cpu_inductor* ]] 2024-06-26T07:10:38.1412255Z + [[ dynamic_inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:10:38.1413659Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-06-26T07:10:38.1414369Z ++ pwd 2024-06-26T07:10:38.1414835Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:38.1415520Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:38.1424697Z + local name=inference 2024-06-26T07:10:38.1425836Z + shift 2024-06-26T07:10:38.1426188Z + local suite=huggingface 2024-06-26T07:10:38.1426545Z + shift 2024-06-26T07:10:38.1426810Z + local shard_id=0 2024-06-26T07:10:38.1427109Z + shift 2024-06-26T07:10:38.1427374Z + partition_flags=() 2024-06-26T07:10:38.1427702Z + local partition_flags 2024-06-26T07:10:38.1428085Z + [[ -n 1 ]] 2024-06-26T07:10:38.1428390Z + [[ -n 0 ]] 2024-06-26T07:10:38.1428966Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:10:38.1429684Z + [[ dynamic_inductor_huggingface == *perf_compare* ]] 2024-06-26T07:10:38.1430343Z + [[ dynamic_inductor_huggingface == *perf* ]] 2024-06-26T07:10:38.1430856Z + [[ dynamic_inductor_huggingface == *aot_inductor* ]] 2024-06-26T07:10:38.1432695Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-06-26T07:10:40.7801549Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:10:40.7803263Z warnings.warn( 2024-06-26T07:10:40.8238508Z 2024-06-26T07:10:40.8239070Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-06-26T07:36:51.2896642Z pass 2024-06-26T07:36:51.3225177Z TIMING: entire_frame_compile:26.65711 code_gen:13.71043 inductor_compile:22.88398 backend_compile:23.36328 2024-06-26T07:36:51.3226488Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:60169 | FakeTensor.__torch_dispatch__:9058 | ProxyTorchDispatchMode.__torch_dispatch__:19976 2024-06-26T07:36:51.3227605Z Dynamo produced 2 graphs covering 534 ops with 5 graph breaks (4 unique) 2024-06-26T07:36:56.1621801Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:56.1623506Z warnings.warn( 2024-06-26T07:36:56.3546279Z 2024-06-26T07:36:59.0332678Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:59.0333297Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:36:59.0333837Z cuda train AllenaiLongformerBase 2024-06-26T07:37:07.1881281Z W0626 07:37:07.187000 140554524213888 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T07:37:07.1882697Z W0626 07:37:07.187000 140554524213888 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T07:37:07.1883841Z W0626 07:37:07.187000 140554524213888 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T07:37:07.1884879Z W0626 07:37:07.187000 140554524213888 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T07:37:07.1886133Z W0626 07:37:07.187000 140554524213888 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T07:37:07.1887178Z W0626 07:37:07.187000 140554524213888 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T07:38:28.2621167Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:38:28.2623558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in torch_dynamo_resume_in_forward_at_1291 2024-06-26T07:38:28.2624703Z layer_outputs = layer_module( 2024-06-26T07:38:28.2625760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:38:28.2626606Z return forward_call(*args, **kwargs) 2024-06-26T07:38:28.2627689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-06-26T07:38:28.2628664Z self_attn_outputs = self.attention( 2024-06-26T07:38:28.2633357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:38:28.2634715Z return forward_call(*args, **kwargs) 2024-06-26T07:38:28.2636008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-06-26T07:38:28.2637145Z self_outputs = self.self( 2024-06-26T07:38:28.2638026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:38:28.2638867Z return forward_call(*args, **kwargs) 2024-06-26T07:38:28.2639915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 571, in forward 2024-06-26T07:38:28.2640923Z attn_scores = self._sliding_chunks_query_key_matmul( 2024-06-26T07:38:28.2642171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 836, in _sliding_chunks_query_key_matmul 2024-06-26T07:38:28.2643459Z query = self._chunk(query, window_overlap, getattr(self.config, "onnx_export", False)) 2024-06-26T07:38:28.2644722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 778, in _chunk 2024-06-26T07:38:28.2645831Z return hidden_states.as_strided(size=chunk_size, stride=chunk_stride) 2024-06-26T07:38:28.2646277Z 2024-06-26T07:38:28.8078660Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:38:28.8080134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1734, in forward 2024-06-26T07:38:28.8081121Z embedding_output = self.embeddings( 2024-06-26T07:38:28.8082054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:38:28.8082880Z return forward_call(*args, **kwargs) 2024-06-26T07:38:28.8083928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 470, in forward 2024-06-26T07:38:28.8084922Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:38:28.8085278Z 2024-06-26T07:38:28.8147464Z W0626 07:38:28.814000 140554524213888 torch/_logging/_internal.py:1040] [16/0] Profiler function will be ignored 2024-06-26T07:39:26.1333506Z pass 2024-06-26T07:39:26.1820479Z TIMING: entire_frame_compile:106.55518 inductor_compile:74.13764 backend_compile:90.64376 code_gen:38.94778 2024-06-26T07:39:26.1823107Z STATS: call_* op count: 2367 | FakeTensorMode.__torch_dispatch__:186737 | FakeTensor.__torch_dispatch__:21002 | ProxyTorchDispatchMode.__torch_dispatch__:58074 2024-06-26T07:39:26.1825365Z Dynamo produced 7 graphs covering 2367 ops with 9 graph breaks (5 unique) 2024-06-26T07:39:34.9810954Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:34.9812821Z warnings.warn( 2024-06-26T07:39:35.1737064Z 2024-06-26T07:39:38.7928630Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:38.7929106Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:39:38.7929550Z cuda train BartForCausalLM 2024-06-26T07:40:01.4212789Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:40:01.4214338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T07:40:01.4217294Z return super().forward(positions + self.offset) 2024-06-26T07:40:01.4218455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:40:01.4219254Z return F.embedding( 2024-06-26T07:40:01.4219472Z 2024-06-26T07:40:08.1216535Z pass 2024-06-26T07:40:08.2447536Z TIMING: entire_frame_compile:12.3367 inductor_compile:4.287 backend_compile:10.74901 code_gen:2.04191 2024-06-26T07:40:08.2448834Z STATS: call_* op count: 325 | FakeTensorMode.__torch_dispatch__:32936 | FakeTensor.__torch_dispatch__:2534 | ProxyTorchDispatchMode.__torch_dispatch__:15974 2024-06-26T07:40:08.2449985Z Dynamo produced 16 graphs covering 325 ops with 6 graph breaks (5 unique) 2024-06-26T07:40:12.0279437Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:12.0281080Z warnings.warn( 2024-06-26T07:40:12.2271250Z 2024-06-26T07:40:18.7468798Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:18.7469537Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:40:18.7470388Z cuda train BartForConditionalGeneration 2024-06-26T07:41:04.7519388Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:41:04.7520916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T07:41:04.7524623Z return super().forward(positions + self.offset) 2024-06-26T07:41:04.7526629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:41:04.7527800Z return F.embedding( 2024-06-26T07:41:04.7528020Z 2024-06-26T07:41:06.1530827Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:41:06.1532026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T07:41:06.1532987Z return super().forward(positions + self.offset) 2024-06-26T07:41:06.1533919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:41:06.1534706Z return F.embedding( 2024-06-26T07:41:06.1534935Z 2024-06-26T07:41:26.8246470Z pass 2024-06-26T07:41:27.0257265Z TIMING: entire_frame_compile:26.98522 inductor_compile:7.63612 backend_compile:23.13467 code_gen:2.9513 2024-06-26T07:41:27.0260021Z STATS: call_* op count: 878 | FakeTensorMode.__torch_dispatch__:83431 | FakeTensor.__torch_dispatch__:6224 | ProxyTorchDispatchMode.__torch_dispatch__:41429 2024-06-26T07:41:27.0262252Z Dynamo produced 30 graphs covering 878 ops with 8 graph breaks (5 unique) 2024-06-26T07:41:31.2997250Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:41:31.2999052Z warnings.warn( 2024-06-26T07:41:31.4771122Z 2024-06-26T07:41:33.3202558Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:41:33.3203028Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:41:33.3205832Z cuda train BertForMaskedLM 2024-06-26T07:42:08.7958853Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:42:08.7960924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:42:08.7962375Z pred = mod(**cloned_inputs) 2024-06-26T07:42:08.7963678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:42:08.7964514Z return forward_call(*args, **kwargs) 2024-06-26T07:42:08.7965522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-06-26T07:42:08.7966400Z outputs = self.bert( 2024-06-26T07:42:08.7967244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:42:08.7968073Z return forward_call(*args, **kwargs) 2024-06-26T07:42:08.7973901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T07:42:08.7974872Z embedding_output = self.embeddings( 2024-06-26T07:42:08.7975818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:42:08.7976636Z return forward_call(*args, **kwargs) 2024-06-26T07:42:08.7977622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T07:42:08.7978549Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:42:08.7978888Z 2024-06-26T07:42:08.8785348Z W0626 07:42:08.877000 140690805265024 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:42:58.1485910Z pass 2024-06-26T07:42:58.2115350Z TIMING: entire_frame_compile:68.71086 code_gen:23.43512 inductor_compile:43.52903 backend_compile:56.85024 2024-06-26T07:42:58.2117757Z STATS: call_* op count: 996 | FakeTensorMode.__torch_dispatch__:109856 | FakeTensor.__torch_dispatch__:18352 | ProxyTorchDispatchMode.__torch_dispatch__:26951 2024-06-26T07:42:58.2118957Z Dynamo produced 2 graphs covering 996 ops with 5 graph breaks (4 unique) 2024-06-26T07:43:05.3032512Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:43:05.3034231Z warnings.warn( 2024-06-26T07:43:05.4949018Z 2024-06-26T07:43:07.1197798Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:43:07.1198277Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:43:07.1198746Z cuda train BertForQuestionAnswering 2024-06-26T07:43:41.2718645Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:43:41.2719967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:43:41.2721027Z pred = mod(**cloned_inputs) 2024-06-26T07:43:41.2722106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:43:41.2723027Z return forward_call(*args, **kwargs) 2024-06-26T07:43:41.2724169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1846, in forward 2024-06-26T07:43:41.2725217Z outputs = self.bert( 2024-06-26T07:43:41.2726331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:43:41.2727170Z return forward_call(*args, **kwargs) 2024-06-26T07:43:41.2728170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T07:43:41.2729115Z embedding_output = self.embeddings( 2024-06-26T07:43:41.2730054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:43:41.2731086Z return forward_call(*args, **kwargs) 2024-06-26T07:43:41.2732061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T07:43:41.2732992Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:43:41.2733452Z 2024-06-26T07:43:41.3463493Z W0626 07:43:41.345000 139884393173632 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:44:30.1619343Z pass 2024-06-26T07:44:30.2251658Z TIMING: entire_frame_compile:67.78014 code_gen:22.47332 inductor_compile:42.96296 backend_compile:56.21574 2024-06-26T07:44:30.2254142Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:108909 | FakeTensor.__torch_dispatch__:18133 | ProxyTorchDispatchMode.__torch_dispatch__:26764 2024-06-26T07:44:30.2255529Z Dynamo produced 2 graphs covering 994 ops with 5 graph breaks (4 unique) 2024-06-26T07:44:37.2440902Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:44:37.2442582Z warnings.warn( 2024-06-26T07:44:37.4590884Z 2024-06-26T07:45:04.6347991Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:45:04.6351425Z loading model: 0it [00:27, ?it/s] 2024-06-26T07:45:04.6352038Z cuda train BlenderbotForCausalLM 2024-06-26T07:45:04.6353046Z Traceback (most recent call last): 2024-06-26T07:45:04.6353856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T07:45:04.6356035Z self.model_iter_fn(model, example_inputs) 2024-06-26T07:45:04.6357219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:45:04.6358073Z pred = mod(**cloned_inputs) 2024-06-26T07:45:04.6359119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:45:04.6360138Z return self._call_impl(*args, **kwargs) 2024-06-26T07:45:04.6361523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:45:04.6362677Z return forward_call(*args, **kwargs) 2024-06-26T07:45:04.6364150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-06-26T07:45:04.6365631Z outputs = self.model.decoder( 2024-06-26T07:45:04.6366640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:45:04.6367533Z return self._call_impl(*args, **kwargs) 2024-06-26T07:45:04.6368462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:45:04.6369297Z return forward_call(*args, **kwargs) 2024-06-26T07:45:04.6370372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-06-26T07:45:04.6371327Z layer_outputs = decoder_layer( 2024-06-26T07:45:04.6372276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:45:04.6373168Z return self._call_impl(*args, **kwargs) 2024-06-26T07:45:04.6374084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:45:04.6374918Z return forward_call(*args, **kwargs) 2024-06-26T07:45:04.6375973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-06-26T07:45:04.6377078Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-06-26T07:45:04.6378464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:45:04.6379357Z return self._call_impl(*args, **kwargs) 2024-06-26T07:45:04.6380379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:45:04.6381213Z return forward_call(*args, **kwargs) 2024-06-26T07:45:04.6382267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-06-26T07:45:04.6383318Z query_states = self.q_proj(hidden_states) * self.scaling 2024-06-26T07:45:04.6384373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:45:04.6385275Z return self._call_impl(*args, **kwargs) 2024-06-26T07:45:04.6386248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:45:04.6387093Z return forward_call(*args, **kwargs) 2024-06-26T07:45:04.6387992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-06-26T07:45:04.6388849Z return F.linear(input, self.weight, self.bias) 2024-06-26T07:45:04.6392251Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 21981 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:45:04.6395238Z 2024-06-26T07:45:04.6395537Z The above exception was the direct cause of the following exception: 2024-06-26T07:45:04.6395985Z 2024-06-26T07:45:04.6396138Z Traceback (most recent call last): 2024-06-26T07:45:04.6396783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T07:45:04.6397433Z ) = runner.load_model( 2024-06-26T07:45:04.6398099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 522, in load_model 2024-06-26T07:45:04.6398857Z self.validate_model(model, example_inputs) 2024-06-26T07:45:04.6399606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T07:45:04.6400357Z raise RuntimeError("Eager run failed") from e 2024-06-26T07:45:04.6400824Z RuntimeError: Eager run failed 2024-06-26T07:45:04.6401077Z 2024-06-26T07:45:04.6401199Z eager_fail_to_run 2024-06-26T07:45:07.9398745Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:45:07.9400454Z warnings.warn( 2024-06-26T07:45:08.1315565Z 2024-06-26T07:45:09.3903480Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:45:09.3904064Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:45:09.3904555Z cuda train BlenderbotSmallForCausalLM 2024-06-26T07:45:09.4116676Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-06-26T07:45:23.7069000Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:45:23.7070574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-06-26T07:45:23.7071597Z return super().forward(positions) 2024-06-26T07:45:23.7072488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:45:23.7073527Z return F.embedding( 2024-06-26T07:45:23.7073748Z 2024-06-26T07:45:28.2575335Z pass 2024-06-26T07:45:28.2857251Z TIMING: entire_frame_compile:10.5251 inductor_compile:4.70294 backend_compile:9.00698 code_gen:2.67686 2024-06-26T07:45:28.2860198Z STATS: call_* op count: 324 | FakeTensorMode.__torch_dispatch__:25848 | FakeTensor.__torch_dispatch__:1921 | ProxyTorchDispatchMode.__torch_dispatch__:12278 2024-06-26T07:45:28.2862489Z Dynamo produced 13 graphs covering 324 ops with 6 graph breaks (5 unique) 2024-06-26T07:45:32.0840575Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:45:32.0842297Z warnings.warn( 2024-06-26T07:45:32.2755683Z 2024-06-26T07:45:33.9381320Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:45:33.9381787Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:45:33.9382321Z cuda train BlenderbotSmallForConditionalGeneration 2024-06-26T07:45:34.0772812Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:45:59.5841938Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:45:59.5845677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-06-26T07:45:59.5846839Z return super().forward(positions) 2024-06-26T07:45:59.5847811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:45:59.5848661Z return F.embedding( 2024-06-26T07:45:59.5848881Z 2024-06-26T07:46:00.7306228Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:46:00.7308196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-06-26T07:46:00.7309440Z return super().forward(positions) 2024-06-26T07:46:00.7310552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T07:46:00.7311350Z return F.embedding( 2024-06-26T07:46:00.7311588Z 2024-06-26T07:46:12.7949160Z pass 2024-06-26T07:46:12.8251289Z TIMING: entire_frame_compile:21.45043 inductor_compile:7.12317 backend_compile:18.03984 code_gen:2.65159 2024-06-26T07:46:12.8252582Z STATS: call_* op count: 830 | FakeTensorMode.__torch_dispatch__:64219 | FakeTensor.__torch_dispatch__:4523 | ProxyTorchDispatchMode.__torch_dispatch__:31624 2024-06-26T07:46:12.8253723Z Dynamo produced 22 graphs covering 830 ops with 8 graph breaks (5 unique) 2024-06-26T07:46:16.9301775Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:46:16.9303488Z warnings.warn( 2024-06-26T07:46:17.1895239Z 2024-06-26T07:46:19.0505272Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:19.0506166Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:46:19.0507072Z cuda train CamemBert 2024-06-26T07:46:54.5731332Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:46:54.5732664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:46:54.5733752Z pred = mod(**cloned_inputs) 2024-06-26T07:46:54.5734843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:54.5735761Z return forward_call(*args, **kwargs) 2024-06-26T07:46:54.5737043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 979, in forward 2024-06-26T07:46:54.5738321Z outputs = self.roberta( 2024-06-26T07:46:54.5739221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:54.5740059Z return forward_call(*args, **kwargs) 2024-06-26T07:46:54.5741243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 881, in forward 2024-06-26T07:46:54.5742203Z embedding_output = self.embeddings( 2024-06-26T07:46:54.5743134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:46:54.5743966Z return forward_call(*args, **kwargs) 2024-06-26T07:46:54.5745012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 139, in forward 2024-06-26T07:46:54.5746023Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:46:54.5746379Z 2024-06-26T07:46:54.8727113Z W0626 07:46:54.871000 140329689440896 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:47:44.1923682Z pass 2024-06-26T07:47:44.2557868Z TIMING: entire_frame_compile:68.70399 code_gen:23.19202 inductor_compile:43.39682 backend_compile:56.89439 2024-06-26T07:47:44.2559193Z STATS: call_* op count: 1004 | FakeTensorMode.__torch_dispatch__:110009 | FakeTensor.__torch_dispatch__:18376 | ProxyTorchDispatchMode.__torch_dispatch__:26999 2024-06-26T07:47:44.2560338Z Dynamo produced 2 graphs covering 1004 ops with 5 graph breaks (4 unique) 2024-06-26T07:47:51.3557398Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:47:51.5738359Z warnings.warn( 2024-06-26T07:47:51.5738622Z 2024-06-26T07:47:53.7865637Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:47:53.7866252Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:47:53.7866828Z cuda train DebertaForMaskedLM 2024-06-26T07:48:31.8373142Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:48:31.8374411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:48:31.8378218Z pred = mod(**cloned_inputs) 2024-06-26T07:48:31.8379513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:48:31.8380707Z return forward_call(*args, **kwargs) 2024-06-26T07:48:31.8382093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1062, in forward 2024-06-26T07:48:31.8383172Z outputs = self.deberta( 2024-06-26T07:48:31.8384067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:48:31.8384909Z return forward_call(*args, **kwargs) 2024-06-26T07:48:31.8386043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-06-26T07:48:31.8387000Z embedding_output = self.embeddings( 2024-06-26T07:48:31.8387930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:48:31.8388785Z return forward_call(*args, **kwargs) 2024-06-26T07:48:31.8389806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-06-26T07:48:31.8390903Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:48:31.8391265Z 2024-06-26T07:48:32.1459656Z W0626 07:48:32.144000 139823584060032 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:49:13.7027740Z pass 2024-06-26T07:49:13.7615022Z TIMING: entire_frame_compile:63.10561 code_gen:22.47384 inductor_compile:41.34677 backend_compile:52.32588 2024-06-26T07:49:13.7618013Z STATS: call_* op count: 1319 | FakeTensorMode.__torch_dispatch__:103763 | FakeTensor.__torch_dispatch__:16054 | ProxyTorchDispatchMode.__torch_dispatch__:28168 2024-06-26T07:49:13.7619184Z Dynamo produced 2 graphs covering 1319 ops with 5 graph breaks (4 unique) 2024-06-26T07:49:20.5001606Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:49:20.5003305Z warnings.warn( 2024-06-26T07:49:20.6916499Z 2024-06-26T07:49:22.5083382Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:22.5083914Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:49:22.5084377Z cuda train DebertaForQuestionAnswering 2024-06-26T07:49:57.7167843Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:49:57.7169241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:49:57.7170809Z pred = mod(**cloned_inputs) 2024-06-26T07:49:57.7171863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:49:57.7172739Z return forward_call(*args, **kwargs) 2024-06-26T07:49:57.7174006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1388, in forward 2024-06-26T07:49:57.7175167Z outputs = self.deberta( 2024-06-26T07:49:57.7176032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:49:57.7177019Z return forward_call(*args, **kwargs) 2024-06-26T07:49:57.7178361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-06-26T07:49:57.7179304Z embedding_output = self.embeddings( 2024-06-26T07:49:57.7180226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:49:57.7181055Z return forward_call(*args, **kwargs) 2024-06-26T07:49:57.7182111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-06-26T07:49:57.7183067Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:49:57.7183416Z 2024-06-26T07:49:57.7882143Z W0626 07:49:57.787000 140492903060096 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:50:36.6553675Z pass 2024-06-26T07:50:36.7221390Z TIMING: entire_frame_compile:58.8909 code_gen:19.46059 inductor_compile:37.97335 backend_compile:48.31265 2024-06-26T07:50:36.7224418Z STATS: call_* op count: 1317 | FakeTensorMode.__torch_dispatch__:102777 | FakeTensor.__torch_dispatch__:15836 | ProxyTorchDispatchMode.__torch_dispatch__:27972 2024-06-26T07:50:36.7225617Z Dynamo produced 2 graphs covering 1317 ops with 5 graph breaks (4 unique) 2024-06-26T07:50:43.4070378Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:50:43.4072016Z warnings.warn( 2024-06-26T07:50:43.6631289Z 2024-06-26T07:50:55.6157751Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:50:55.6158375Z loading model: 0it [00:11, ?it/s] 2024-06-26T07:50:55.6158993Z cuda train DebertaV2ForMaskedLM 2024-06-26T07:50:55.6159992Z pass_due_to_skip 2024-06-26T07:50:55.8364131Z TIMING: 2024-06-26T07:50:55.8364553Z STATS: call_* op count: 0 2024-06-26T07:50:55.8365184Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:50:59.1119376Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:50:59.1122121Z warnings.warn( 2024-06-26T07:50:59.3293677Z 2024-06-26T07:51:09.2603934Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:09.2604652Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:51:09.2605349Z cuda train DebertaV2ForQuestionAnswering 2024-06-26T07:51:19.8874329Z ERROR:common: 2024-06-26T07:51:19.8875099Z Traceback (most recent call last): 2024-06-26T07:51:19.8877054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2543, in check_accuracy 2024-06-26T07:51:19.8879096Z correct_result = self.run_n_iterations( 2024-06-26T07:51:19.8880243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T07:51:19.8881464Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T07:51:19.8882764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:51:19.8883920Z pred = mod(**cloned_inputs) 2024-06-26T07:51:19.8885210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:51:19.8886104Z return self._call_impl(*args, **kwargs) 2024-06-26T07:51:19.8887344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:19.8888403Z return forward_call(*args, **kwargs) 2024-06-26T07:51:19.8889693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-06-26T07:51:19.8890844Z outputs = self.deberta( 2024-06-26T07:51:19.8891764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:51:19.8892646Z return self._call_impl(*args, **kwargs) 2024-06-26T07:51:19.8893568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:19.8894394Z return forward_call(*args, **kwargs) 2024-06-26T07:51:19.8895433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-06-26T07:51:19.8896357Z encoder_outputs = self.encoder( 2024-06-26T07:51:19.8897313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:51:19.8898202Z return self._call_impl(*args, **kwargs) 2024-06-26T07:51:19.8899180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:19.8900003Z return forward_call(*args, **kwargs) 2024-06-26T07:51:19.8901042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-06-26T07:51:19.8901969Z output_states = layer_module( 2024-06-26T07:51:19.8902899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:51:19.8903767Z return self._call_impl(*args, **kwargs) 2024-06-26T07:51:19.8904674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:19.8905501Z return forward_call(*args, **kwargs) 2024-06-26T07:51:19.8906543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-06-26T07:51:19.8907698Z attention_output = self.attention( 2024-06-26T07:51:19.8908653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:51:19.8909534Z return self._call_impl(*args, **kwargs) 2024-06-26T07:51:19.8910729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:19.8911573Z return forward_call(*args, **kwargs) 2024-06-26T07:51:19.8912614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-06-26T07:51:19.8913554Z self_output = self.self( 2024-06-26T07:51:19.8914688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:51:19.8915595Z return self._call_impl(*args, **kwargs) 2024-06-26T07:51:19.8916521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:19.8917362Z return forward_call(*args, **kwargs) 2024-06-26T07:51:19.8918413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-06-26T07:51:19.8919578Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-06-26T07:51:19.8920609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply 2024-06-26T07:51:19.8921505Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-06-26T07:51:19.8922638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-06-26T07:51:19.8923607Z output = torch.softmax(output, self.dim) 2024-06-26T07:51:19.8926809Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 23520 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:51:19.9197871Z eager_1st_run_OOM 2024-06-26T07:51:19.9198308Z TIMING: 2024-06-26T07:51:19.9198849Z STATS: call_* op count: 0 2024-06-26T07:51:19.9199537Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:51:23.1894845Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:51:23.1896708Z warnings.warn( 2024-06-26T07:51:23.6154595Z 2024-06-26T07:51:24.9153776Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:24.9154254Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:51:24.9154944Z cuda train DistilBertForMaskedLM 2024-06-26T07:51:43.2197207Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:51:43.2198535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:51:43.2199497Z pred = mod(**cloned_inputs) 2024-06-26T07:51:43.2200622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:43.2201768Z return forward_call(*args, **kwargs) 2024-06-26T07:51:43.2205665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-06-26T07:51:43.2206844Z dlbrt_output = self.distilbert( 2024-06-26T07:51:43.2208068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:43.2208990Z return forward_call(*args, **kwargs) 2024-06-26T07:51:43.2210290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-06-26T07:51:43.2211531Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-06-26T07:51:43.2212786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:43.2213701Z return forward_call(*args, **kwargs) 2024-06-26T07:51:43.2214826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-06-26T07:51:43.2216059Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-06-26T07:51:43.2216617Z 2024-06-26T07:51:43.3800484Z W0626 07:51:43.379000 139848628572800 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:52:10.5137853Z pass 2024-06-26T07:52:10.5509348Z TIMING: entire_frame_compile:37.42331 code_gen:13.8937 inductor_compile:24.64031 backend_compile:32.02776 2024-06-26T07:52:10.5514094Z STATS: call_* op count: 541 | FakeTensor.__torch_dispatch__:9534 | FakeTensorMode.__torch_dispatch__:58009 | ProxyTorchDispatchMode.__torch_dispatch__:14211 2024-06-26T07:52:10.5515858Z Dynamo produced 2 graphs covering 541 ops with 5 graph breaks (4 unique) 2024-06-26T07:52:15.8362244Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:15.8363952Z warnings.warn( 2024-06-26T07:52:16.0230606Z 2024-06-26T07:52:17.1053681Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:17.1054152Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:52:17.1054610Z cuda train DistilBertForQuestionAnswering 2024-06-26T07:52:34.4984840Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:52:34.4986069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:52:34.4987034Z pred = mod(**cloned_inputs) 2024-06-26T07:52:34.4988976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:34.4990115Z return forward_call(*args, **kwargs) 2024-06-26T07:52:34.4991324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1124, in forward 2024-06-26T07:52:34.4992298Z distilbert_output = self.distilbert( 2024-06-26T07:52:34.4993563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:34.4994966Z return forward_call(*args, **kwargs) 2024-06-26T07:52:34.4996030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-06-26T07:52:34.4997166Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-06-26T07:52:34.4998269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:52:34.4999102Z return forward_call(*args, **kwargs) 2024-06-26T07:52:34.5000147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-06-26T07:52:34.5001263Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-06-26T07:52:34.5001730Z 2024-06-26T07:52:34.5626382Z W0626 07:52:34.561000 140695878623872 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:52:57.8427261Z pass 2024-06-26T07:52:57.8802846Z TIMING: entire_frame_compile:33.16587 code_gen:9.741 inductor_compile:20.25997 backend_compile:27.7141 2024-06-26T07:52:57.8805188Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:9327 | FakeTensorMode.__torch_dispatch__:57125 | ProxyTorchDispatchMode.__torch_dispatch__:14037 2024-06-26T07:52:57.8806352Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-06-26T07:53:03.0443685Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:53:03.0445346Z warnings.warn( 2024-06-26T07:53:03.4464954Z 2024-06-26T07:53:05.2334324Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:05.2334842Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:53:05.2339012Z cuda train DistillGPT2 2024-06-26T07:53:24.1838782Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:53:24.1840637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:53:24.1842148Z pred = mod(**cloned_inputs) 2024-06-26T07:53:24.1843529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:24.1844543Z return forward_call(*args, **kwargs) 2024-06-26T07:53:24.1845583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-06-26T07:53:24.1846510Z transformer_outputs = self.transformer( 2024-06-26T07:53:24.1847457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:53:24.1848304Z return forward_call(*args, **kwargs) 2024-06-26T07:53:24.1849288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-06-26T07:53:24.1850185Z inputs_embeds = self.wte(input_ids) 2024-06-26T07:53:24.1850482Z 2024-06-26T07:53:24.2440574Z W0626 07:53:24.243000 140518040752768 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:53:45.5925956Z pass 2024-06-26T07:53:45.6279074Z TIMING: entire_frame_compile:31.35071 code_gen:12.64528 inductor_compile:21.616 backend_compile:26.29595 2024-06-26T07:53:45.6281397Z STATS: call_* op count: 572 | FakeTensorMode.__torch_dispatch__:46229 | FakeTensor.__torch_dispatch__:7295 | ProxyTorchDispatchMode.__torch_dispatch__:11477 2024-06-26T07:53:45.6282613Z Dynamo produced 2 graphs covering 572 ops with 5 graph breaks (4 unique) 2024-06-26T07:53:50.5383039Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:53:50.5384779Z warnings.warn( 2024-06-26T07:53:50.7328029Z 2024-06-26T07:53:50.7336722Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:53:51.3366023Z 2024-06-26T07:53:51.3366463Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:51.3369500Z cuda train ElectraForCausalLM 2024-06-26T07:54:24.9595081Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:54:24.9597590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:54:24.9598907Z pred = mod(**cloned_inputs) 2024-06-26T07:54:24.9600589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:54:24.9601675Z return forward_call(*args, **kwargs) 2024-06-26T07:54:24.9602996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1617, in forward 2024-06-26T07:54:24.9604097Z outputs = self.electra( 2024-06-26T07:54:24.9604967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:54:24.9605783Z return forward_call(*args, **kwargs) 2024-06-26T07:54:24.9606802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-06-26T07:54:24.9607716Z hidden_states = self.embeddings( 2024-06-26T07:54:24.9608642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:54:24.9609506Z return forward_call(*args, **kwargs) 2024-06-26T07:54:24.9610505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-06-26T07:54:24.9611457Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:54:24.9611801Z 2024-06-26T07:54:25.6719498Z pass 2024-06-26T07:54:25.6903660Z TIMING: entire_frame_compile:21.9775 code_gen:8.54748 inductor_compile:16.75443 backend_compile:17.2431 2024-06-26T07:54:25.6905054Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:56448 | FakeTensor.__torch_dispatch__:8074 | ProxyTorchDispatchMode.__torch_dispatch__:17988 2024-06-26T07:54:25.6906285Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-06-26T07:54:30.5102896Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:54:30.5104581Z warnings.warn( 2024-06-26T07:54:30.7065984Z 2024-06-26T07:54:31.2908074Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:54:31.2908555Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:54:31.2909028Z cuda train ElectraForQuestionAnswering 2024-06-26T07:55:04.0720930Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:55:04.0723248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:55:04.0724401Z pred = mod(**cloned_inputs) 2024-06-26T07:55:04.0725783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:55:04.0726940Z return forward_call(*args, **kwargs) 2024-06-26T07:55:04.0728187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1385, in forward 2024-06-26T07:55:04.0729482Z discriminator_hidden_states = self.electra( 2024-06-26T07:55:04.0730517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:55:04.0731432Z return forward_call(*args, **kwargs) 2024-06-26T07:55:04.0732548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-06-26T07:55:04.0733706Z hidden_states = self.embeddings( 2024-06-26T07:55:04.0734627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:55:04.0735458Z return forward_call(*args, **kwargs) 2024-06-26T07:55:04.0736461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-06-26T07:55:04.0737424Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:55:04.0738038Z 2024-06-26T07:55:04.1489519Z W0626 07:55:04.148000 140048498274944 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:55:53.5544822Z pass 2024-06-26T07:55:53.5685526Z TIMING: entire_frame_compile:68.45921 code_gen:22.70477 inductor_compile:42.80025 backend_compile:56.82446 2024-06-26T07:55:53.5688011Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:109856 | FakeTensor.__torch_dispatch__:18301 | ProxyTorchDispatchMode.__torch_dispatch__:26961 2024-06-26T07:55:53.5689325Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:56:00.7229374Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:56:00.7231186Z warnings.warn( 2024-06-26T07:56:00.9402239Z 2024-06-26T07:56:03.0016360Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:56:03.0016881Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:56:03.0017402Z cuda train GPT2ForSequenceClassification 2024-06-26T07:56:38.7026646Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:56:38.7027984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1426, in forward 2024-06-26T07:56:38.7029123Z transformer_outputs = self.transformer( 2024-06-26T07:56:38.7030677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:56:38.7031624Z return forward_call(*args, **kwargs) 2024-06-26T07:56:38.7034220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-06-26T07:56:38.7035383Z inputs_embeds = self.wte(input_ids) 2024-06-26T07:56:38.7035685Z 2024-06-26T07:56:38.7436837Z W0626 07:56:38.742000 139944246444672 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:57:17.1749359Z pass 2024-06-26T07:57:17.2528945Z TIMING: entire_frame_compile:56.14598 inductor_compile:36.38862 backend_compile:45.40619 code_gen:19.51685 2024-06-26T07:57:17.2531154Z STATS: call_* op count: 1102 | FakeTensorMode.__torch_dispatch__:93384 | FakeTensor.__torch_dispatch__:14173 | ProxyTorchDispatchMode.__torch_dispatch__:22505 2024-06-26T07:57:17.2532327Z Dynamo produced 5 graphs covering 1102 ops with 7 graph breaks (5 unique) 2024-06-26T07:57:23.5492241Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:57:23.5493873Z warnings.warn( 2024-06-26T07:57:23.7387506Z 2024-06-26T07:57:25.0357194Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:57:25.1926715Z 2024-06-26T07:57:25.1927578Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:57:25.1928173Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:57:25.1938434Z cuda train GoogleFnet 2024-06-26T07:57:26.5221247Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:57:43.6330815Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:57:43.6331982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:57:43.6334444Z pred = mod(**cloned_inputs) 2024-06-26T07:57:43.6335711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:57:43.6336562Z return forward_call(*args, **kwargs) 2024-06-26T07:57:43.6337830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 749, in forward 2024-06-26T07:57:43.6339235Z outputs = self.fnet( 2024-06-26T07:57:43.6340100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:57:43.6341043Z return forward_call(*args, **kwargs) 2024-06-26T07:57:43.6342027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 581, in forward 2024-06-26T07:57:43.6342931Z embedding_output = self.embeddings( 2024-06-26T07:57:43.6343850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:57:43.6344678Z return forward_call(*args, **kwargs) 2024-06-26T07:57:43.6345655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 148, in forward 2024-06-26T07:57:43.6346589Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:57:43.6346948Z 2024-06-26T07:57:43.7005470Z W0626 07:57:43.699000 140313434464896 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:58:11.2669438Z pass 2024-06-26T07:58:11.3085169Z TIMING: entire_frame_compile:36.25255 code_gen:14.56994 inductor_compile:25.707 backend_compile:30.14457 2024-06-26T07:58:11.3087007Z STATS: call_* op count: 574 | FakeTensorMode.__torch_dispatch__:52210 | FakeTensor.__torch_dispatch__:9591 | ProxyTorchDispatchMode.__torch_dispatch__:12282 2024-06-26T07:58:11.3088156Z Dynamo produced 2 graphs covering 574 ops with 5 graph breaks (4 unique) 2024-06-26T07:58:16.6124217Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:58:16.6125916Z warnings.warn( 2024-06-26T07:58:16.8632539Z 2024-06-26T07:58:19.0330737Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:58:19.0331242Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:58:19.0331687Z cuda train LayoutLMForMaskedLM 2024-06-26T07:58:54.7527676Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:58:54.7528862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T07:58:54.7530000Z pred = mod(**cloned_inputs) 2024-06-26T07:58:54.7531028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:58:54.7531879Z return forward_call(*args, **kwargs) 2024-06-26T07:58:54.7532923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 938, in forward 2024-06-26T07:58:54.7533860Z outputs = self.layoutlm( 2024-06-26T07:58:54.7534729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:58:54.7535658Z return forward_call(*args, **kwargs) 2024-06-26T07:58:54.7537018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-06-26T07:58:54.7538063Z embedding_output = self.embeddings( 2024-06-26T07:58:54.7538982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:58:54.7539804Z return forward_call(*args, **kwargs) 2024-06-26T07:58:54.7540817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-06-26T07:58:54.7541788Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T07:58:54.7542132Z 2024-06-26T07:58:54.8373214Z W0626 07:58:54.836000 139738484925056 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:59:45.0803542Z pass 2024-06-26T07:59:45.1415827Z TIMING: entire_frame_compile:70.21573 code_gen:24.1619 inductor_compile:44.23892 backend_compile:57.99442 2024-06-26T07:59:45.1417364Z STATS: call_* op count: 1034 | FakeTensorMode.__torch_dispatch__:112112 | FakeTensor.__torch_dispatch__:18659 | ProxyTorchDispatchMode.__torch_dispatch__:27517 2024-06-26T07:59:45.1422039Z Dynamo produced 2 graphs covering 1034 ops with 5 graph breaks (4 unique) 2024-06-26T07:59:52.2994975Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:59:52.2996678Z warnings.warn( 2024-06-26T07:59:52.5175435Z 2024-06-26T07:59:54.1508372Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:59:54.1509288Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:59:54.1510388Z cuda train LayoutLMForSequenceClassification 2024-06-26T08:00:30.8438069Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:00:30.8440318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 1060, in forward 2024-06-26T08:00:30.8441373Z outputs = self.layoutlm( 2024-06-26T08:00:30.8442654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:00:30.8443782Z return forward_call(*args, **kwargs) 2024-06-26T08:00:30.8446458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-06-26T08:00:30.8447564Z embedding_output = self.embeddings( 2024-06-26T08:00:30.8448527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:00:30.8449378Z return forward_call(*args, **kwargs) 2024-06-26T08:00:30.8450423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-06-26T08:00:30.8451404Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:00:30.8451756Z 2024-06-26T08:00:30.9264688Z W0626 08:00:30.925000 140692435665536 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T08:01:19.9011734Z pass 2024-06-26T08:01:19.9684236Z TIMING: entire_frame_compile:68.73467 inductor_compile:40.42598 backend_compile:54.49294 code_gen:19.88434 2024-06-26T08:01:19.9686178Z STATS: call_* op count: 1031 | FakeTensorMode.__torch_dispatch__:117000 | FakeTensor.__torch_dispatch__:18852 | ProxyTorchDispatchMode.__torch_dispatch__:27791 2024-06-26T08:01:19.9687370Z Dynamo produced 5 graphs covering 1031 ops with 7 graph breaks (5 unique) 2024-06-26T08:01:27.0901486Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:01:27.0903151Z warnings.warn( 2024-06-26T08:01:27.2955663Z 2024-06-26T08:01:37.2602839Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:37.2604361Z loading model: 0it [00:09, ?it/s] 2024-06-26T08:01:37.2605010Z cuda train M2M100ForConditionalGeneration 2024-06-26T08:01:37.3395884Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:02:46.1074281Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:02:46.1076177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:02:46.1077396Z pred = mod(**cloned_inputs) 2024-06-26T08:02:46.1078388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:02:46.1079222Z return forward_call(*args, **kwargs) 2024-06-26T08:02:46.1080365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1275, in forward 2024-06-26T08:02:46.1081582Z outputs = self.model( 2024-06-26T08:02:46.1082665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:02:46.1083752Z return forward_call(*args, **kwargs) 2024-06-26T08:02:46.1084859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1162, in forward 2024-06-26T08:02:46.1085769Z encoder_outputs = self.encoder( 2024-06-26T08:02:46.1086672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:02:46.1087512Z return forward_call(*args, **kwargs) 2024-06-26T08:02:46.1088519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 763, in forward 2024-06-26T08:02:46.1089545Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-06-26T08:02:46.1089973Z 2024-06-26T08:02:47.7514958Z pass 2024-06-26T08:02:47.8459219Z TIMING: entire_frame_compile:45.44392 code_gen:19.43672 inductor_compile:39.67057 backend_compile:39.9116 2024-06-26T08:02:47.8461644Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:140959 | FakeTensor.__torch_dispatch__:17728 | ProxyTorchDispatchMode.__torch_dispatch__:47005 2024-06-26T08:02:47.8463146Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-06-26T08:02:54.4110130Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:02:54.4112144Z warnings.warn( 2024-06-26T08:02:54.6229695Z 2024-06-26T08:02:58.2508859Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:02:58.2509562Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:02:58.2510411Z cuda train MBartForCausalLM 2024-06-26T08:02:58.2820549Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-06-26T08:03:14.7868169Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:03:14.7869592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T08:03:14.7870871Z return super().forward(positions + self.offset) 2024-06-26T08:03:14.7872040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:03:14.7872876Z return F.embedding( 2024-06-26T08:03:14.7873101Z 2024-06-26T08:03:21.9077266Z pass 2024-06-26T08:03:21.9744646Z TIMING: entire_frame_compile:12.80087 inductor_compile:4.19686 backend_compile:10.96197 code_gen:1.75742 2024-06-26T08:03:21.9745996Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:36729 | FakeTensor.__torch_dispatch__:2464 | ProxyTorchDispatchMode.__torch_dispatch__:18211 2024-06-26T08:03:21.9747131Z Dynamo produced 17 graphs covering 478 ops with 6 graph breaks (5 unique) 2024-06-26T08:03:25.8006775Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:03:25.8008518Z warnings.warn( 2024-06-26T08:03:25.9828460Z 2024-06-26T08:03:32.5807753Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:03:32.5808666Z loading model: 0it [00:06, ?it/s] 2024-06-26T08:03:32.5810006Z cuda train MBartForConditionalGeneration 2024-06-26T08:03:32.6809315Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:04:08.6704750Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:04:08.6706981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T08:04:08.6708364Z return super().forward(positions + self.offset) 2024-06-26T08:04:08.6709755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:04:08.6711005Z return F.embedding( 2024-06-26T08:04:08.6711302Z 2024-06-26T08:04:10.6768110Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:04:10.6769560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-06-26T08:04:10.6770533Z return super().forward(positions + self.offset) 2024-06-26T08:04:10.6771469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:04:10.6772243Z return F.embedding( 2024-06-26T08:04:10.6772469Z 2024-06-26T08:04:32.6804202Z pass 2024-06-26T08:04:32.7839103Z TIMING: entire_frame_compile:29.74776 inductor_compile:8.57307 backend_compile:25.50957 code_gen:3.09971 2024-06-26T08:04:32.7841915Z STATS: call_* op count: 1251 | FakeTensorMode.__torch_dispatch__:93472 | FakeTensor.__torch_dispatch__:6086 | ProxyTorchDispatchMode.__torch_dispatch__:47299 2024-06-26T08:04:32.7843098Z Dynamo produced 31 graphs covering 1251 ops with 8 graph breaks (5 unique) 2024-06-26T08:04:37.1857188Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:04:37.1858914Z warnings.warn( 2024-06-26T08:04:37.5500924Z 2024-06-26T08:04:41.3787805Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:04:41.3788685Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:04:41.3789333Z cuda train MT5ForConditionalGeneration 2024-06-26T08:05:36.0308592Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:05:36.0309772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:05:36.0310853Z pred = mod(**cloned_inputs) 2024-06-26T08:05:36.0312111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:05:36.0313141Z return forward_call(*args, **kwargs) 2024-06-26T08:05:36.0314736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1722, in forward 2024-06-26T08:05:36.0315822Z encoder_outputs = self.encoder( 2024-06-26T08:05:36.0316808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:05:36.0317712Z return forward_call(*args, **kwargs) 2024-06-26T08:05:36.0318778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 989, in forward 2024-06-26T08:05:36.0319753Z inputs_embeds = self.embed_tokens(input_ids) 2024-06-26T08:05:36.0320087Z 2024-06-26T08:05:36.1202349Z W0626 08:05:36.119000 140207001150080 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:06:27.8501833Z pass 2024-06-26T08:06:27.9939586Z TIMING: entire_frame_compile:83.36771 code_gen:26.88196 inductor_compile:52.88408 backend_compile:66.41735 2024-06-26T08:06:27.9942873Z STATS: call_* op count: 1764 | FakeTensorMode.__torch_dispatch__:155529 | FakeTensor.__torch_dispatch__:22500 | ProxyTorchDispatchMode.__torch_dispatch__:44560 2024-06-26T08:06:27.9945008Z Dynamo produced 2 graphs covering 1764 ops with 5 graph breaks (4 unique) 2024-06-26T08:06:35.7027803Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:06:35.7029435Z warnings.warn( 2024-06-26T08:06:35.8955593Z 2024-06-26T08:06:35.9002630Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T08:06:40.3358185Z 2024-06-26T08:06:40.3358874Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:06:40.3359607Z cuda train MegatronBertForCausalLM 2024-06-26T08:07:48.3035083Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:07:48.3037694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:07:48.3038765Z pred = mod(**cloned_inputs) 2024-06-26T08:07:48.3040057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:48.3041223Z return forward_call(*args, **kwargs) 2024-06-26T08:07:48.3042776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1199, in forward 2024-06-26T08:07:48.3044031Z outputs = self.bert( 2024-06-26T08:07:48.3045183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:48.3046041Z return forward_call(*args, **kwargs) 2024-06-26T08:07:48.3047147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-06-26T08:07:48.3103938Z embedding_output = self.embeddings( 2024-06-26T08:07:48.3105160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:48.3106005Z return forward_call(*args, **kwargs) 2024-06-26T08:07:48.3107092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-06-26T08:07:48.3108109Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:07:48.3108464Z 2024-06-26T08:07:48.4144674Z W0626 08:07:48.413000 139824913244800 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:09:23.9031517Z pass 2024-06-26T08:09:24.0984821Z TIMING: entire_frame_compile:131.67819 code_gen:40.7382 inductor_compile:81.68047 backend_compile:107.54916 2024-06-26T08:09:24.0986120Z STATS: call_* op count: 1923 | FakeTensorMode.__torch_dispatch__:213841 | FakeTensor.__torch_dispatch__:35697 | ProxyTorchDispatchMode.__torch_dispatch__:52645 2024-06-26T08:09:24.0988211Z Dynamo produced 2 graphs covering 1923 ops with 5 graph breaks (4 unique) 2024-06-26T08:09:34.8201772Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:09:34.8203494Z warnings.warn( 2024-06-26T08:09:34.9934729Z 2024-06-26T08:09:39.1341343Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:09:39.1341803Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:09:39.1346181Z cuda train MegatronBertForQuestionAnswering 2024-06-26T08:10:46.0991021Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:10:46.0993271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:10:46.0995119Z pred = mod(**cloned_inputs) 2024-06-26T08:10:46.0996275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:10:46.0997445Z return forward_call(*args, **kwargs) 2024-06-26T08:10:46.0998991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1792, in forward 2024-06-26T08:10:46.1000008Z outputs = self.bert( 2024-06-26T08:10:46.1000851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:10:46.1001674Z return forward_call(*args, **kwargs) 2024-06-26T08:10:46.1002879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-06-26T08:10:46.1003856Z embedding_output = self.embeddings( 2024-06-26T08:10:46.1004758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:10:46.1005602Z return forward_call(*args, **kwargs) 2024-06-26T08:10:46.1006671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-06-26T08:10:46.1007692Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:10:46.1008039Z 2024-06-26T08:10:46.2003589Z W0626 08:10:46.199000 140406830338688 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:12:18.0855968Z pass 2024-06-26T08:12:18.2824850Z TIMING: entire_frame_compile:127.82578 code_gen:37.94095 inductor_compile:77.82501 backend_compile:104.00708 2024-06-26T08:12:18.2827448Z STATS: call_* op count: 1917 | FakeTensorMode.__torch_dispatch__:212755 | FakeTensor.__torch_dispatch__:35470 | ProxyTorchDispatchMode.__torch_dispatch__:52414 2024-06-26T08:12:18.2828845Z Dynamo produced 2 graphs covering 1917 ops with 5 graph breaks (4 unique) 2024-06-26T08:12:28.9775664Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:12:28.9777363Z warnings.warn( 2024-06-26T08:12:29.2034801Z 2024-06-26T08:12:30.5164607Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:30.5165085Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:12:30.5165540Z cuda train MobileBertForMaskedLM 2024-06-26T08:14:32.0883377Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:14:32.0885323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:14:32.0886311Z pred = mod(**cloned_inputs) 2024-06-26T08:14:32.0887323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:14:32.0888346Z return forward_call(*args, **kwargs) 2024-06-26T08:14:32.0889665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1089, in forward 2024-06-26T08:14:32.0890873Z outputs = self.mobilebert( 2024-06-26T08:14:32.0892065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:14:32.0892905Z return forward_call(*args, **kwargs) 2024-06-26T08:14:32.0893978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-06-26T08:14:32.0894942Z embedding_output = self.embeddings( 2024-06-26T08:14:32.0895863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:14:32.0896911Z return forward_call(*args, **kwargs) 2024-06-26T08:14:32.0897960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-06-26T08:14:32.0898962Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:14:32.0899309Z 2024-06-26T08:14:34.6730769Z pass 2024-06-26T08:14:34.7064203Z TIMING: entire_frame_compile:84.76295 code_gen:27.45471 inductor_compile:56.85397 backend_compile:61.26961 2024-06-26T08:14:34.7066627Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:216936 | FakeTensor.__torch_dispatch__:27386 | ProxyTorchDispatchMode.__torch_dispatch__:74460 2024-06-26T08:14:34.7068727Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-06-26T08:14:43.3364446Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:14:43.3366186Z warnings.warn( 2024-06-26T08:14:43.5801853Z 2024-06-26T08:14:44.7418669Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:44.7419293Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:14:44.7419869Z cuda train MobileBertForQuestionAnswering 2024-06-26T08:16:45.6670819Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:16:45.6672005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:16:45.6675506Z pred = mod(**cloned_inputs) 2024-06-26T08:16:45.6676674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:16:45.6677777Z return forward_call(*args, **kwargs) 2024-06-26T08:16:45.6679091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1390, in forward 2024-06-26T08:16:45.6680172Z outputs = self.mobilebert( 2024-06-26T08:16:45.6681046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:16:45.6682180Z return forward_call(*args, **kwargs) 2024-06-26T08:16:45.6683329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-06-26T08:16:45.6684291Z embedding_output = self.embeddings( 2024-06-26T08:16:45.6685206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:16:45.6686045Z return forward_call(*args, **kwargs) 2024-06-26T08:16:45.6687085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-06-26T08:16:45.6688065Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:16:45.6688425Z 2024-06-26T08:16:48.1364660Z pass 2024-06-26T08:16:48.1583220Z TIMING: entire_frame_compile:84.36587 code_gen:26.75094 inductor_compile:56.19888 backend_compile:60.84738 2024-06-26T08:16:48.1584603Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:216724 | FakeTensor.__torch_dispatch__:27353 | ProxyTorchDispatchMode.__torch_dispatch__:74366 2024-06-26T08:16:48.1585736Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-06-26T08:16:56.7531467Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:16:56.7534537Z warnings.warn( 2024-06-26T08:16:56.9473133Z 2024-06-26T08:16:59.2464657Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:16:59.2465531Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:16:59.2466422Z cuda train OPTForCausalLM 2024-06-26T08:16:59.2661411Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-06-26T08:17:17.4158541Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:17:17.4160205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 109, in forward 2024-06-26T08:17:17.4161473Z return super().forward(positions + self.offset) 2024-06-26T08:17:17.4162445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:17:17.4163233Z return F.embedding( 2024-06-26T08:17:17.4163453Z 2024-06-26T08:17:24.7282971Z pass 2024-06-26T08:17:24.7518220Z TIMING: entire_frame_compile:13.14642 inductor_compile:5.21274 backend_compile:11.35187 code_gen:2.76418 2024-06-26T08:17:28.6090705Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:37247 | FakeTensor.__torch_dispatch__:2544 | ProxyTorchDispatchMode.__torch_dispatch__:18427 2024-06-26T08:17:28.6091939Z Dynamo produced 17 graphs covering 533 ops with 6 graph breaks (5 unique) 2024-06-26T08:17:28.6094074Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:17:28.6095778Z warnings.warn( 2024-06-26T08:17:28.8592730Z 2024-06-26T08:17:30.5986169Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:17:30.5987049Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:17:30.5987723Z cuda train PLBartForCausalLM 2024-06-26T08:17:30.6150702Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-06-26T08:17:43.1062982Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:17:43.1064528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-06-26T08:17:43.1065522Z return super().forward(positions + self.offset) 2024-06-26T08:17:43.1066458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:17:43.1067265Z return F.embedding( 2024-06-26T08:17:43.1067486Z 2024-06-26T08:17:46.5911372Z pass 2024-06-26T08:17:46.6216314Z TIMING: entire_frame_compile:9.08308 inductor_compile:4.3085 backend_compile:7.7725 code_gen:1.97144 2024-06-26T08:17:46.6218852Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:20561 | FakeTensor.__torch_dispatch__:1633 | ProxyTorchDispatchMode.__torch_dispatch__:9427 2024-06-26T08:17:46.6221105Z Dynamo produced 11 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-06-26T08:17:50.4195961Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:17:50.4197743Z warnings.warn( 2024-06-26T08:17:50.6148841Z 2024-06-26T08:17:53.7569584Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:17:53.7570454Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:17:53.7571425Z cuda train PLBartForConditionalGeneration 2024-06-26T08:17:53.8093593Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:18:16.4929836Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:18:16.4931655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-06-26T08:18:16.4934334Z return super().forward(positions + self.offset) 2024-06-26T08:18:16.4935542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:18:16.4936664Z return F.embedding( 2024-06-26T08:18:16.4936886Z 2024-06-26T08:18:17.6959071Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:18:17.6960952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-06-26T08:18:17.6962683Z return super().forward(positions + self.offset) 2024-06-26T08:18:17.6963662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:18:17.6964458Z return F.embedding( 2024-06-26T08:18:17.6964692Z 2024-06-26T08:18:26.9203349Z pass 2024-06-26T08:18:26.9563044Z TIMING: entire_frame_compile:17.99074 inductor_compile:7.62046 backend_compile:15.32583 code_gen:3.26024 2024-06-26T08:18:26.9564347Z STATS: call_* op count: 646 | FakeTensorMode.__torch_dispatch__:51723 | FakeTensor.__torch_dispatch__:3926 | ProxyTorchDispatchMode.__torch_dispatch__:24571 2024-06-26T08:18:26.9565501Z Dynamo produced 19 graphs covering 646 ops with 8 graph breaks (5 unique) 2024-06-26T08:18:31.0416913Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:18:31.0418579Z warnings.warn( 2024-06-26T08:18:31.2557851Z 2024-06-26T08:18:37.2649368Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:18:37.2649855Z loading model: 0it [00:06, ?it/s] 2024-06-26T08:18:37.2650314Z cuda train PegasusForCausalLM 2024-06-26T08:18:37.2955962Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-06-26T08:19:01.9952050Z pass 2024-06-26T08:19:02.0747875Z TIMING: entire_frame_compile:13.89068 inductor_compile:4.64491 backend_compile:11.65278 code_gen:1.6321 2024-06-26T08:19:02.0749454Z STATS: call_* op count: 476 | FakeTensorMode.__torch_dispatch__:36892 | FakeTensor.__torch_dispatch__:2548 | ProxyTorchDispatchMode.__torch_dispatch__:18176 2024-06-26T08:19:02.0752896Z Dynamo produced 17 graphs covering 476 ops with 6 graph breaks (5 unique) 2024-06-26T08:19:05.8920271Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:19:05.8922057Z warnings.warn( 2024-06-26T08:19:06.1182929Z 2024-06-26T08:19:16.3332639Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:19:16.3333130Z loading model: 0it [00:10, ?it/s] 2024-06-26T08:19:16.3333605Z cuda train PegasusForConditionalGeneration 2024-06-26T08:19:16.4094980Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T08:20:14.1377094Z pass 2024-06-26T08:20:14.2209712Z TIMING: entire_frame_compile:29.32468 inductor_compile:8.24818 backend_compile:25.14813 code_gen:2.73614 2024-06-26T08:20:14.2211665Z STATS: call_* op count: 1234 | FakeTensorMode.__torch_dispatch__:93033 | FakeTensor.__torch_dispatch__:6119 | ProxyTorchDispatchMode.__torch_dispatch__:47120 2024-06-26T08:20:14.2212866Z Dynamo produced 30 graphs covering 1234 ops with 7 graph breaks (4 unique) 2024-06-26T08:20:18.5837315Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:20:18.5838957Z warnings.warn( 2024-06-26T08:20:18.7664099Z 2024-06-26T08:20:18.7698976Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-06-26T08:20:20.6709776Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T08:20:20.6712257Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T08:20:20.8899438Z 2024-06-26T08:20:20.8899901Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:20:20.8900448Z cuda train RobertaForCausalLM 2024-06-26T08:20:55.5780816Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:20:55.5782109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:20:55.5785683Z pred = mod(**cloned_inputs) 2024-06-26T08:20:55.5787487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:20:55.5788874Z return forward_call(*args, **kwargs) 2024-06-26T08:20:55.5790482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 953, in forward 2024-06-26T08:20:55.5791874Z outputs = self.roberta( 2024-06-26T08:20:55.5793150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:20:55.5794673Z return forward_call(*args, **kwargs) 2024-06-26T08:20:55.5796229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-06-26T08:20:55.5797622Z embedding_output = self.embeddings( 2024-06-26T08:20:55.5799270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:20:55.5800759Z return forward_call(*args, **kwargs) 2024-06-26T08:20:55.5802234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-06-26T08:20:55.5803618Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:20:55.5804170Z 2024-06-26T08:20:55.7635272Z W0626 08:20:55.762000 139723663663744 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:21:45.0437001Z pass 2024-06-26T08:21:45.0951694Z TIMING: entire_frame_compile:67.89211 code_gen:22.51563 inductor_compile:42.82368 backend_compile:56.13709 2024-06-26T08:21:45.0953671Z STATS: call_* op count: 1008 | FakeTensorMode.__torch_dispatch__:110136 | FakeTensor.__torch_dispatch__:18383 | ProxyTorchDispatchMode.__torch_dispatch__:27039 2024-06-26T08:21:45.0955180Z Dynamo produced 2 graphs covering 1008 ops with 5 graph breaks (4 unique) 2024-06-26T08:21:52.2609834Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:21:52.2611680Z warnings.warn( 2024-06-26T08:21:52.4499242Z 2024-06-26T08:21:53.9917643Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T08:21:53.9919789Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T08:21:54.2311376Z 2024-06-26T08:21:54.2312044Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:21:54.2312584Z cuda train RobertaForQuestionAnswering 2024-06-26T08:22:27.8796280Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:22:27.8798196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:22:27.8799462Z pred = mod(**cloned_inputs) 2024-06-26T08:22:27.8800467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:27.8801445Z return forward_call(*args, **kwargs) 2024-06-26T08:22:27.8802889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1500, in forward 2024-06-26T08:22:27.8803828Z outputs = self.roberta( 2024-06-26T08:22:27.8804940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:27.8805976Z return forward_call(*args, **kwargs) 2024-06-26T08:22:27.8807153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-06-26T08:22:27.8808092Z embedding_output = self.embeddings( 2024-06-26T08:22:27.8809024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:27.8809858Z return forward_call(*args, **kwargs) 2024-06-26T08:22:27.8810875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-06-26T08:22:27.8811856Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:22:27.8812214Z 2024-06-26T08:22:27.9559729Z W0626 08:22:27.955000 139967200653952 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:23:13.3866045Z pass 2024-06-26T08:23:13.4493026Z TIMING: entire_frame_compile:63.83407 code_gen:18.92318 inductor_compile:39.4636 backend_compile:52.25013 2024-06-26T08:23:13.4495420Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:109017 | FakeTensor.__torch_dispatch__:18152 | ProxyTorchDispatchMode.__torch_dispatch__:26799 2024-06-26T08:23:13.4496738Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T08:23:20.5260508Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:23:20.5262678Z warnings.warn( 2024-06-26T08:23:20.7795182Z 2024-06-26T08:23:21.3059861Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:23:21.3060750Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:23:21.3061435Z cuda train Speech2Text2ForCausalLM 2024-06-26T08:23:21.3217883Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-06-26T08:23:35.7451119Z pass 2024-06-26T08:23:35.7499165Z TIMING: entire_frame_compile:8.24212 inductor_compile:4.24308 backend_compile:7.0982 code_gen:1.8269 2024-06-26T08:23:35.7500459Z STATS: call_* op count: 258 | FakeTensorMode.__torch_dispatch__:20223 | FakeTensor.__torch_dispatch__:1555 | ProxyTorchDispatchMode.__torch_dispatch__:9305 2024-06-26T08:23:35.7501594Z Dynamo produced 11 graphs covering 258 ops with 6 graph breaks (5 unique) 2024-06-26T08:23:39.5180532Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:23:39.5182231Z warnings.warn( 2024-06-26T08:23:39.7362351Z 2024-06-26T08:23:40.9777594Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:23:40.9778085Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:23:40.9778598Z cuda train T5ForConditionalGeneration 2024-06-26T08:24:20.2520096Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:24:20.2524223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:24:20.2525972Z pred = mod(**cloned_inputs) 2024-06-26T08:24:20.2527108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:24:20.2527959Z return forward_call(*args, **kwargs) 2024-06-26T08:24:20.2529063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-06-26T08:24:20.2529942Z encoder_outputs = self.encoder( 2024-06-26T08:24:20.2532571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:24:20.2533588Z return forward_call(*args, **kwargs) 2024-06-26T08:24:20.2534585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-06-26T08:24:20.2535496Z inputs_embeds = self.embed_tokens(input_ids) 2024-06-26T08:24:20.2535852Z 2024-06-26T08:24:20.3226241Z W0626 08:24:20.321000 140438486385280 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:24:55.7744067Z pass 2024-06-26T08:24:55.8346358Z TIMING: entire_frame_compile:57.70198 code_gen:18.9446 inductor_compile:36.80407 backend_compile:47.27509 2024-06-26T08:24:55.8347976Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:107352 | FakeTensor.__torch_dispatch__:14731 | ProxyTorchDispatchMode.__torch_dispatch__:30407 2024-06-26T08:24:55.8351557Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T08:25:02.1686822Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:25:02.1689389Z warnings.warn( 2024-06-26T08:25:02.4082509Z 2024-06-26T08:25:03.6541672Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:25:03.6542560Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:25:03.6543415Z cuda train T5Small 2024-06-26T08:25:26.8386997Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:25:26.8388202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:25:26.8389669Z pred = mod(**cloned_inputs) 2024-06-26T08:25:26.8390820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:25:26.8391665Z return forward_call(*args, **kwargs) 2024-06-26T08:25:26.8392634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-06-26T08:25:26.8393510Z encoder_outputs = self.encoder( 2024-06-26T08:25:26.8394711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:25:26.8395861Z return forward_call(*args, **kwargs) 2024-06-26T08:25:26.8396887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-06-26T08:25:26.8399073Z inputs_embeds = self.embed_tokens(input_ids) 2024-06-26T08:25:26.8399482Z 2024-06-26T08:25:26.9075977Z W0626 08:25:26.906000 139693692465792 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:25:44.4957265Z pass 2024-06-26T08:25:44.5548123Z TIMING: entire_frame_compile:33.7169 inductor_compile:3.85808 backend_compile:23.28182 2024-06-26T08:25:44.5549647Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:71205 | FakeTensor.__torch_dispatch__:4747 | ProxyTorchDispatchMode.__torch_dispatch__:30407 2024-06-26T08:25:44.5550931Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T08:25:48.8047704Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:25:48.8049723Z warnings.warn( 2024-06-26T08:25:49.0792364Z 2024-06-26T08:25:52.7058166Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:25:52.7058648Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:25:52.7059094Z cuda train TrOCRForCausalLM 2024-06-26T08:25:52.7366222Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-06-26T08:26:10.5295656Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:26:10.5296961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 66, in forward 2024-06-26T08:26:10.5299043Z return super().forward(positions + self.offset) 2024-06-26T08:26:10.5300228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T08:26:10.5301085Z return F.embedding( 2024-06-26T08:26:10.5301303Z 2024-06-26T08:26:17.7782997Z pass 2024-06-26T08:26:17.8627820Z TIMING: entire_frame_compile:13.90145 inductor_compile:4.94792 backend_compile:11.86814 code_gen:2.55105 2024-06-26T08:26:17.8629720Z STATS: call_* op count: 477 | FakeTensorMode.__torch_dispatch__:37042 | FakeTensor.__torch_dispatch__:2529 | ProxyTorchDispatchMode.__torch_dispatch__:18211 2024-06-26T08:26:17.8631518Z Dynamo produced 17 graphs covering 477 ops with 6 graph breaks (5 unique) 2024-06-26T08:26:21.7509571Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:26:21.7512326Z warnings.warn( 2024-06-26T08:26:21.9353537Z 2024-06-26T08:26:31.1492895Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:26:31.1493489Z loading model: 0it [00:09, ?it/s] 2024-06-26T08:26:31.1495963Z cuda train XGLMForCausalLM 2024-06-26T08:26:31.2941346Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-06-26T08:26:35.1954792Z skipping cudagraphs due to mutated inputs (1 instances). Found from : 2024-06-26T08:26:35.1956411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 174, in forward 2024-06-26T08:26:35.1957289Z position_ids += self.offset 2024-06-26T08:26:35.1957557Z 2024-06-26T08:27:18.5039684Z pass 2024-06-26T08:27:18.5864503Z TIMING: entire_frame_compile:24.18712 inductor_compile:6.16019 backend_compile:20.96136 code_gen:1.95944 2024-06-26T08:27:18.5865995Z STATS: call_* op count: 987 | FakeTensorMode.__torch_dispatch__:72969 | FakeTensor.__torch_dispatch__:4507 | ProxyTorchDispatchMode.__torch_dispatch__:37344 2024-06-26T08:27:18.5867156Z Dynamo produced 29 graphs covering 987 ops with 6 graph breaks (5 unique) 2024-06-26T08:27:22.7245040Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:27:22.7247622Z warnings.warn( 2024-06-26T08:27:22.9026846Z 2024-06-26T08:27:27.0820966Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:27:27.0821600Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:27:27.0824323Z cuda train XLNetLMHeadModel 2024-06-26T08:28:21.2398463Z W0626 08:28:21.239000 140270409618048 torch/_inductor/utils.py:1234] [3/0_1] DeviceCopy in input program 2024-06-26T08:28:34.6939545Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-06-26T08:28:34.6941728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:28:34.6943223Z pred = mod(**cloned_inputs) 2024-06-26T08:28:34.6949877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:34.6951453Z return forward_call(*args, **kwargs) 2024-06-26T08:28:34.6953124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1446, in forward 2024-06-26T08:28:34.6954999Z transformer_outputs = self.transformer( 2024-06-26T08:28:34.6956557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:28:34.6957922Z return forward_call(*args, **kwargs) 2024-06-26T08:28:34.6959573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1203, in forward 2024-06-26T08:28:34.6961293Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2024-06-26T08:28:34.6963349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1055, in relative_positional_encoding 2024-06-26T08:28:34.6965238Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2024-06-26T08:28:34.6967220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1014, in positional_embedding 2024-06-26T08:28:34.6968827Z pos_emb = pos_emb[:, None, :] 2024-06-26T08:28:34.6969259Z 2024-06-26T08:28:58.3399257Z W0626 08:28:58.339000 140270409618048 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:30:31.4469192Z pass 2024-06-26T08:30:31.6500699Z TIMING: entire_frame_compile:147.55443 code_gen:42.67107 inductor_compile:88.87837 backend_compile:121.36856 2024-06-26T08:30:31.6502349Z STATS: call_* op count: 1874 | FakeTensorMode.__torch_dispatch__:262068 | FakeTensor.__torch_dispatch__:35162 | ProxyTorchDispatchMode.__torch_dispatch__:66724 2024-06-26T08:30:31.6511871Z Dynamo produced 2 graphs covering 1874 ops with 5 graph breaks (4 unique) 2024-06-26T08:30:42.5715958Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T08:30:42.5717634Z warnings.warn( 2024-06-26T08:30:42.7792273Z 2024-06-26T08:30:44.8302879Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:44.8303348Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:30:44.8303789Z cuda train YituTechConvBert 2024-06-26T08:31:37.8999283Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:31:37.9000497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in torch_dynamo_resume_in_forward_and_backward_pass_at_580 2024-06-26T08:31:37.9001452Z pred = mod(**cloned_inputs) 2024-06-26T08:31:37.9002646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:37.9005184Z return forward_call(*args, **kwargs) 2024-06-26T08:31:37.9006534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 918, in forward 2024-06-26T08:31:37.9007627Z generator_hidden_states = self.convbert( 2024-06-26T08:31:37.9008720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:37.9009605Z return forward_call(*args, **kwargs) 2024-06-26T08:31:37.9010779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 834, in forward 2024-06-26T08:31:37.9012042Z hidden_states = self.embeddings( 2024-06-26T08:31:37.9013060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:37.9013964Z return forward_call(*args, **kwargs) 2024-06-26T08:31:37.9015196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 230, in forward 2024-06-26T08:31:37.9018010Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:31:37.9018476Z 2024-06-26T08:31:37.9985608Z W0626 08:31:37.997000 139884563583616 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T08:32:51.0083830Z pass 2024-06-26T08:32:51.0621669Z TIMING: entire_frame_compile:102.73218 code_gen:36.03947 inductor_compile:65.91722 backend_compile:84.51327 2024-06-26T08:32:51.0625479Z STATS: call_* op count: 1512 | FakeTensorMode.__torch_dispatch__:161410 | FakeTensor.__torch_dispatch__:25959 | ProxyTorchDispatchMode.__torch_dispatch__:39878 2024-06-26T08:32:51.0626679Z Dynamo produced 2 graphs covering 1512 ops with 5 graph breaks (4 unique) 2024-06-26T08:32:57.4015606Z accuracy pass_rate=93.48% 2024-06-26T08:32:57.4019215Z calls_captured gmean=0.00x mean=920.348x 2024-06-26T08:32:57.4023406Z unique_graphs gmean=0.00x mean=7.478x 2024-06-26T08:32:57.4027374Z graph_breaks gmean=0.00x mean=5.196x 2024-06-26T08:32:57.4031546Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-06-26T08:32:57.4035534Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:32:57.4039480Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:32:57.4042703Z cudagraph_skips gmean=0.00x mean=0.978x 2024-06-26T08:32:58.2463737Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_huggingface_training.csv 2024-06-26T08:32:58.4853837Z AlbertForMaskedLM PASS 2024-06-26T08:32:58.4858281Z AlbertForQuestionAnswering PASS 2024-06-26T08:32:58.4862608Z AllenaiLongformerBase PASS 2024-06-26T08:32:58.4867343Z BartForCausalLM PASS 2024-06-26T08:32:58.4871876Z BartForConditionalGeneration PASS 2024-06-26T08:32:58.4876306Z BertForMaskedLM PASS 2024-06-26T08:32:58.4880773Z BertForQuestionAnswering PASS 2024-06-26T08:32:58.4885156Z BlenderbotForCausalLM XFAIL 2024-06-26T08:32:58.4889989Z BlenderbotSmallForCausalLM PASS 2024-06-26T08:32:58.4894574Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T08:32:58.4899202Z CamemBert PASS 2024-06-26T08:32:58.4903838Z DebertaForMaskedLM PASS 2024-06-26T08:32:58.4908265Z DebertaForQuestionAnswering PASS 2024-06-26T08:32:58.4913460Z DebertaV2ForMaskedLM XFAIL 2024-06-26T08:32:58.4918202Z DebertaV2ForQuestionAnswering XFAIL 2024-06-26T08:32:58.4923130Z DistilBertForMaskedLM PASS 2024-06-26T08:32:58.4927776Z DistilBertForQuestionAnswering PASS 2024-06-26T08:32:58.4932179Z DistillGPT2 PASS 2024-06-26T08:32:58.4936651Z ElectraForCausalLM PASS 2024-06-26T08:32:58.4941215Z ElectraForQuestionAnswering PASS 2024-06-26T08:32:58.4945961Z GPT2ForSequenceClassification PASS 2024-06-26T08:32:58.4950624Z GoogleFnet PASS 2024-06-26T08:32:58.4955862Z LayoutLMForMaskedLM PASS 2024-06-26T08:32:58.4960344Z LayoutLMForSequenceClassification PASS 2024-06-26T08:32:58.4964907Z M2M100ForConditionalGeneration PASS 2024-06-26T08:32:58.4969368Z MBartForCausalLM PASS 2024-06-26T08:32:58.4973953Z MBartForConditionalGeneration PASS 2024-06-26T08:32:58.4978413Z MT5ForConditionalGeneration PASS 2024-06-26T08:32:58.4983034Z MegatronBertForCausalLM PASS 2024-06-26T08:32:58.4987554Z MegatronBertForQuestionAnswering PASS 2024-06-26T08:32:58.4992328Z MobileBertForMaskedLM PASS 2024-06-26T08:32:58.4997096Z MobileBertForQuestionAnswering PASS 2024-06-26T08:32:58.5001760Z OPTForCausalLM PASS 2024-06-26T08:32:58.5006429Z PLBartForCausalLM PASS 2024-06-26T08:32:58.5011032Z PLBartForConditionalGeneration PASS 2024-06-26T08:32:58.5015422Z PegasusForCausalLM PASS 2024-06-26T08:32:58.5019933Z PegasusForConditionalGeneration PASS 2024-06-26T08:32:58.5024346Z RobertaForCausalLM PASS 2024-06-26T08:32:58.5028834Z RobertaForQuestionAnswering PASS 2024-06-26T08:32:58.5033700Z Speech2Text2ForCausalLM PASS 2024-06-26T08:32:58.5038382Z T5ForConditionalGeneration PASS 2024-06-26T08:32:58.5042738Z T5Small PASS 2024-06-26T08:32:58.5047248Z TrOCRForCausalLM PASS 2024-06-26T08:32:58.5051787Z XGLMForCausalLM PASS 2024-06-26T08:32:58.5056281Z XLNetLMHeadModel PASS 2024-06-26T08:32:58.5060751Z YituTechConvBert PASS 2024-06-26T08:32:58.5452368Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_huggingface_training.csv 2024-06-26T08:32:58.8052118Z AlbertForMaskedLM PASS 2024-06-26T08:32:58.8058577Z AlbertForQuestionAnswering PASS 2024-06-26T08:32:58.8066076Z AllenaiLongformerBase PASS 2024-06-26T08:32:58.8073571Z BartForCausalLM PASS 2024-06-26T08:32:58.8081428Z BartForConditionalGeneration PASS 2024-06-26T08:32:58.8086168Z BertForMaskedLM PASS 2024-06-26T08:32:58.8092230Z BertForQuestionAnswering PASS 2024-06-26T08:32:58.8096941Z BlenderbotForCausalLM PASS 2024-06-26T08:32:58.8102432Z BlenderbotSmallForCausalLM PASS 2024-06-26T08:32:58.8107252Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T08:32:58.8112841Z CamemBert PASS 2024-06-26T08:32:58.8117760Z DebertaForMaskedLM PASS 2024-06-26T08:32:58.8122838Z DebertaForQuestionAnswering PASS 2024-06-26T08:32:58.8127675Z DebertaV2ForMaskedLM PASS 2024-06-26T08:32:58.8132543Z DebertaV2ForQuestionAnswering PASS 2024-06-26T08:32:58.8137682Z DistilBertForMaskedLM PASS 2024-06-26T08:32:58.8142494Z DistilBertForQuestionAnswering PASS 2024-06-26T08:32:58.8147511Z DistillGPT2 PASS 2024-06-26T08:32:58.8152825Z ElectraForCausalLM PASS 2024-06-26T08:32:58.8157970Z ElectraForQuestionAnswering PASS 2024-06-26T08:32:58.8162856Z GPT2ForSequenceClassification PASS 2024-06-26T08:32:58.8167805Z GoogleFnet PASS 2024-06-26T08:32:58.8172620Z LayoutLMForMaskedLM PASS 2024-06-26T08:32:58.8177706Z LayoutLMForSequenceClassification PASS 2024-06-26T08:32:58.8182531Z M2M100ForConditionalGeneration PASS 2024-06-26T08:32:58.8187388Z MBartForCausalLM PASS 2024-06-26T08:32:58.8192817Z MBartForConditionalGeneration PASS 2024-06-26T08:32:58.8197827Z MT5ForConditionalGeneration PASS 2024-06-26T08:32:58.8202638Z MegatronBertForCausalLM PASS 2024-06-26T08:32:58.8207580Z MegatronBertForQuestionAnswering PASS 2024-06-26T08:32:58.8212409Z MobileBertForMaskedLM PASS 2024-06-26T08:32:58.8217343Z MobileBertForQuestionAnswering PASS 2024-06-26T08:32:58.8222206Z OPTForCausalLM PASS 2024-06-26T08:32:58.8227041Z PLBartForCausalLM PASS 2024-06-26T08:32:58.8232496Z PLBartForConditionalGeneration PASS 2024-06-26T08:32:58.8237499Z PegasusForCausalLM PASS 2024-06-26T08:32:58.8242514Z PegasusForConditionalGeneration PASS 2024-06-26T08:32:58.8247257Z RobertaForCausalLM PASS 2024-06-26T08:32:58.8252272Z RobertaForQuestionAnswering PASS 2024-06-26T08:32:58.8257091Z Speech2Text2ForCausalLM PASS 2024-06-26T08:32:58.8262112Z T5ForConditionalGeneration PASS 2024-06-26T08:32:58.8266880Z T5Small PASS 2024-06-26T08:32:58.8272301Z TrOCRForCausalLM PASS 2024-06-26T08:32:58.8277474Z XGLMForCausalLM PASS 2024-06-26T08:32:58.8282228Z XLNetLMHeadModel PASS 2024-06-26T08:32:58.8287244Z YituTechConvBert PASS 2024-06-26T08:32:58.8666462Z + cleanup_workspace 2024-06-26T08:32:58.8667441Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:32:58.8669858Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:32:58.8671041Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:32:58.8671691Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:32:58.8672512Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:32:58.8673392Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:32:58.8674087Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:32:59.3073966Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:32:59.3074739Z cat test/**/*_toprint.log || true 2024-06-26T08:32:59.3082414Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:32:59.3082914Z env: 2024-06-26T08:32:59.3083195Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.3083640Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.3084368Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.3084999Z ##[endgroup] 2024-06-26T08:32:59.3147384Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:32:59.3319930Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:32:59.3320422Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:32:59.3328031Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:32:59.3328525Z env: 2024-06-26T08:32:59.3328801Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.3329251Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.3329982Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.3330626Z MONITOR_SCRIPT_PID: 9273 2024-06-26T08:32:59.3330975Z ##[endgroup] 2024-06-26T08:32:59.3513820Z Prepare all required actions 2024-06-26T08:32:59.3514276Z Getting action download info 2024-06-26T08:32:59.5438946Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:32:59.6984957Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:32:59.6985426Z with: 2024-06-26T08:32:59.6986007Z file-suffix: test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606 2024-06-26T08:32:59.6986728Z s3-bucket: gha-artifacts 2024-06-26T08:32:59.6987067Z env: 2024-06-26T08:32:59.6987334Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.6987785Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.6988520Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.6989155Z ##[endgroup] 2024-06-26T08:32:59.7013918Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:32:59.7014560Z # Remove any previous test jsons if they exist 2024-06-26T08:32:59.7015076Z rm -f test-jsons-*.zip 2024-06-26T08:32:59.7015594Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:32:59.7023232Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:32:59.7023734Z env: 2024-06-26T08:32:59.7024016Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.7024469Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.7025341Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.7026291Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606 2024-06-26T08:32:59.7026982Z ##[endgroup] 2024-06-26T08:32:59.7201806Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:32:59.7231378Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:32:59.7232356Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:32:59.7238932Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:32:59.7264876Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:32:59.7265516Z # Remove any previous test reports if they exist 2024-06-26T08:32:59.7266043Z rm -f test-reports-*.zip 2024-06-26T08:32:59.7266629Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:32:59.7274676Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:32:59.7275172Z env: 2024-06-26T08:32:59.7275447Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.7275899Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.7276631Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.7277579Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606 2024-06-26T08:32:59.7278269Z ##[endgroup] 2024-06-26T08:32:59.7439935Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-06-26T08:32:59.7440824Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 89%) 2024-06-26T08:32:59.7441773Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 78%) 2024-06-26T08:32:59.7442651Z adding: test/test-reports/training_huggingface.csv (deflated 73%) 2024-06-26T08:32:59.7449270Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-06-26T08:32:59.7450636Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 77%) 2024-06-26T08:32:59.7475844Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:32:59.7476475Z # Remove any previous usage logs if they exist 2024-06-26T08:32:59.7476984Z rm -f logs-*.zip 2024-06-26T08:32:59.7477649Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:32:59.7478553Z # so check to see if the file exists first 2024-06-26T08:32:59.7479058Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:32:59.7479593Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:32:59.7480084Z fi 2024-06-26T08:32:59.7480449Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:32:59.7481020Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:32:59.7481530Z fi 2024-06-26T08:32:59.7488734Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:32:59.7489236Z env: 2024-06-26T08:32:59.7489509Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.7489965Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.7490700Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.7491643Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606 2024-06-26T08:32:59.7492324Z ##[endgroup] 2024-06-26T08:32:59.7563261Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:32:59.7626798Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:32:59.7627555Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:32:59.7628102Z rm -f debug-*.zip 2024-06-26T08:32:59.7628480Z if [ -d 'test/debug' ]; then 2024-06-26T08:32:59.7628984Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:32:59.7629575Z fi 2024-06-26T08:32:59.7636438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:32:59.7636930Z env: 2024-06-26T08:32:59.7637208Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.7637661Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.7638389Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.7639313Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606 2024-06-26T08:32:59.7640006Z ##[endgroup] 2024-06-26T08:32:59.7713759Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:32:59.7714196Z with: 2024-06-26T08:32:59.7714759Z s3-bucket: gha-artifacts 2024-06-26T08:32:59.7715193Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:32:59.7715664Z retention-days: 14 2024-06-26T08:32:59.7715997Z if-no-files-found: warn 2024-06-26T08:32:59.7716355Z path: test-jsons-*.zip 2024-06-26T08:32:59.7716701Z name: artifact 2024-06-26T08:32:59.7716998Z region: us-east-1 2024-06-26T08:32:59.7717346Z env: 2024-06-26T08:32:59.7717617Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:32:59.7718070Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:32:59.7718797Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:32:59.7719441Z ##[endgroup] 2024-06-26T08:33:00.0778521Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:33:00.0779397Z With the provided path, there will be 1 file uploaded 2024-06-26T08:33:00.0780037Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:33:00.0805732Z Starting upload of test-jsons-test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606.zip 2024-06-26T08:33:00.1962353Z Finished upload of test-jsons-test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606.zip 2024-06-26T08:33:00.2092839Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:33:00.2093294Z with: 2024-06-26T08:33:00.2093581Z s3-bucket: gha-artifacts 2024-06-26T08:33:00.2094013Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:33:00.2094486Z retention-days: 14 2024-06-26T08:33:00.2108441Z if-no-files-found: error 2024-06-26T08:33:00.2108828Z path: test-reports-*.zip 2024-06-26T08:33:00.2109177Z name: artifact 2024-06-26T08:33:00.2109488Z region: us-east-1 2024-06-26T08:33:00.2109790Z env: 2024-06-26T08:33:00.2110123Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:00.2110726Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:00.2111469Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:00.2112103Z ##[endgroup] 2024-06-26T08:33:00.5213204Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:33:00.5213823Z With the provided path, there will be 1 file uploaded 2024-06-26T08:33:00.5214435Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:33:00.5239962Z Starting upload of test-reports-test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606.zip 2024-06-26T08:33:00.6258797Z Finished upload of test-reports-test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606.zip 2024-06-26T08:33:00.6389512Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:33:00.6389963Z with: 2024-06-26T08:33:00.6390351Z s3-bucket: gha-artifacts 2024-06-26T08:33:00.6390778Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:33:00.6391256Z retention-days: 14 2024-06-26T08:33:00.6391596Z if-no-files-found: ignore 2024-06-26T08:33:00.6391953Z path: logs-*.zip 2024-06-26T08:33:00.6392258Z name: artifact 2024-06-26T08:33:00.6392555Z region: us-east-1 2024-06-26T08:33:00.6392848Z env: 2024-06-26T08:33:00.6393123Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:00.6393580Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:00.6394318Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:00.6395305Z ##[endgroup] 2024-06-26T08:33:00.9446001Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:33:00.9446900Z With the provided path, there will be 1 file uploaded 2024-06-26T08:33:00.9447743Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:33:00.9474967Z Starting upload of logs-test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606.zip 2024-06-26T08:33:01.0922561Z Finished upload of logs-test-dynamic_inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688911606.zip 2024-06-26T08:33:01.1054416Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:33:01.1054861Z with: 2024-06-26T08:33:01.1055149Z s3-bucket: gha-artifacts 2024-06-26T08:33:01.1055585Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:33:01.1056063Z retention-days: 14 2024-06-26T08:33:01.1056402Z if-no-files-found: ignore 2024-06-26T08:33:01.1056765Z path: debug-*.zip 2024-06-26T08:33:01.1057098Z name: artifact 2024-06-26T08:33:01.1057399Z region: us-east-1 2024-06-26T08:33:01.1057704Z env: 2024-06-26T08:33:01.1057983Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:01.1058444Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:01.1059187Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:01.1059842Z ##[endgroup] 2024-06-26T08:33:01.4043543Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:33:01.4166848Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:33:01.4167332Z # shellcheck disable=SC2156 2024-06-26T08:33:01.4168172Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:33:01.4175966Z shell: /usr/bin/bash -e {0} 2024-06-26T08:33:01.4176321Z env: 2024-06-26T08:33:01.4176601Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:01.4177054Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:01.4177849Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:01.4178499Z ##[endgroup] 2024-06-26T08:33:01.5804101Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:33:01.5804670Z with: 2024-06-26T08:33:01.5804928Z env: 2024-06-26T08:33:01.5805206Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:01.5805655Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:01.5806386Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:01.5807017Z ##[endgroup] 2024-06-26T08:33:01.5824532Z ##[group]Run set -eou pipefail 2024-06-26T08:33:01.5824945Z set -eou pipefail 2024-06-26T08:33:01.5825296Z  2024-06-26T08:33:01.5825805Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:33:01.5826438Z for _ in $(seq 1440); do 2024-06-26T08:33:01.5826926Z  # Break if no ssh session exists anymore 2024-06-26T08:33:01.5827429Z  if [ "$(who)" = "" ]; then 2024-06-26T08:33:01.5827842Z  break 2024-06-26T08:33:01.5828182Z  fi 2024-06-26T08:33:01.5828478Z  echo "." 2024-06-26T08:33:01.5828810Z  sleep 5 2024-06-26T08:33:01.5829128Z done 2024-06-26T08:33:01.5836847Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:01.5837346Z env: 2024-06-26T08:33:01.5837631Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:01.5838085Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:01.5838819Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:01.5839451Z ##[endgroup] 2024-06-26T08:33:01.5859033Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:33:01.5895062Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:33:01.5895947Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:33:01.5896544Z # shellcheck disable=SC2046 2024-06-26T08:33:01.5897017Z docker stop $(docker ps -q) || true 2024-06-26T08:33:01.5897497Z # Prune all of the docker images 2024-06-26T08:33:01.5897956Z docker system prune -af 2024-06-26T08:33:01.5904789Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:01.5905296Z env: 2024-06-26T08:33:01.5905577Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:01.5906033Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:33:01.5906759Z DOCKER_CONTAINER_ID: 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:01.5907389Z ##[endgroup] 2024-06-26T08:33:01.8989168Z 1d5ca5302dc6 2024-06-26T08:33:02.2922994Z Deleted Containers: 2024-06-26T08:33:02.2923624Z 1d5ca5302dc62aab6898e22657160bad045709e5c8b0a93f664f1048c054cef7 2024-06-26T08:33:02.2924055Z 2024-06-26T08:33:07.2754569Z Deleted Images: 2024-06-26T08:33:07.2756377Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:33:07.2759244Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:33:07.2761228Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:33:07.2762285Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:33:07.2763345Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:33:07.2764397Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:33:07.2765462Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:33:07.2766559Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:33:07.2767623Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:33:07.2769205Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:33:07.2770269Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:33:07.2771302Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:33:07.2772349Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:33:07.2773403Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:33:07.2774438Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:33:07.2775484Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:33:07.2776537Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:33:07.2777771Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:33:07.2778826Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:33:07.2779868Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:33:07.2781043Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:33:07.2782108Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:33:07.2783150Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:33:07.2784192Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:33:07.2785223Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:33:07.2786258Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:33:07.2787490Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:33:07.2788544Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:33:07.2789609Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:33:07.2790854Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:33:07.2791915Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:33:07.2792941Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:33:07.2793954Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:33:07.2795312Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:33:07.2796360Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:33:07.2797417Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:33:07.2798482Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:33:07.2799526Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:33:07.2800557Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:33:07.2801585Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:33:07.2802616Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:33:07.2803654Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:33:07.2804700Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:33:07.2805732Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:33:07.2806737Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:33:07.2807776Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:33:07.2808957Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:33:07.2810032Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:33:07.2811070Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:33:07.2812088Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:33:07.2813115Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:33:07.2814167Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:33:07.2815242Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:33:07.2816289Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:33:07.2817355Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:33:07.2818402Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:33:07.2819468Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:33:07.2820559Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:33:07.2821582Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:33:07.2822632Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:33:07.2823674Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:33:07.2824720Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:33:07.2825777Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:33:07.2826955Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:33:07.2828009Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:33:07.2829067Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:33:07.2830257Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:33:07.2831285Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:33:07.2832338Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:33:07.2833374Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:33:07.2834527Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:33:07.2835589Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:33:07.2836620Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:33:07.2837662Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:33:07.2838723Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:33:07.2839792Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:33:07.2840862Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:33:07.2841940Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:33:07.2843038Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:33:07.2844111Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:33:07.2845147Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:33:07.2845781Z 2024-06-26T08:33:07.2845962Z Total reclaimed space: 31.35GB 2024-06-26T08:33:07.2942385Z Post job cleanup. 2024-06-26T08:33:07.2997120Z Post job cleanup. 2024-06-26T08:33:07.3809265Z [command]/usr/bin/git version 2024-06-26T08:33:07.3843591Z git version 2.40.1 2024-06-26T08:33:07.3881596Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e93e7be6-99d1-4053-b292-5a714ee7d965' before making global git config changes 2024-06-26T08:33:07.3882979Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:33:07.3887273Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:33:07.3922638Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:33:07.3951751Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:33:07.4169114Z Entering 'android/libs/fbjni' 2024-06-26T08:33:07.4208850Z Entering 'third_party/FP16' 2024-06-26T08:33:07.4249142Z Entering 'third_party/FXdiv' 2024-06-26T08:33:07.4286364Z Entering 'third_party/NNPACK' 2024-06-26T08:33:07.4324283Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:33:07.4361767Z Entering 'third_party/XNNPACK' 2024-06-26T08:33:07.4410455Z Entering 'third_party/benchmark' 2024-06-26T08:33:07.4449491Z Entering 'third_party/cpp-httplib' 2024-06-26T08:33:07.4484386Z Entering 'third_party/cpuinfo' 2024-06-26T08:33:07.4522238Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:33:07.4559017Z Entering 'third_party/cutlass' 2024-06-26T08:33:07.4599250Z Entering 'third_party/eigen' 2024-06-26T08:33:07.4636988Z Entering 'third_party/fbgemm' 2024-06-26T08:33:07.4675120Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:33:07.4712116Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:33:07.4748921Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:33:07.4790509Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:33:07.4826982Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:33:07.4864732Z Entering 'third_party/flatbuffers' 2024-06-26T08:33:07.4906735Z Entering 'third_party/fmt' 2024-06-26T08:33:07.4943773Z Entering 'third_party/foxi' 2024-06-26T08:33:07.4981389Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:33:07.5018064Z Entering 'third_party/gloo' 2024-06-26T08:33:07.5055345Z Entering 'third_party/googletest' 2024-06-26T08:33:07.5092887Z Entering 'third_party/ideep' 2024-06-26T08:33:07.5127527Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:33:07.5171934Z Entering 'third_party/ittapi' 2024-06-26T08:33:07.5206890Z Entering 'third_party/kineto' 2024-06-26T08:33:07.5242974Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:33:07.5278595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:33:07.5315906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:33:07.5352564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:33:07.5388371Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:33:07.5423843Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:33:07.5461007Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:33:07.5497616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:33:07.5532948Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:33:07.5569833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:33:07.5606188Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:33:07.5641028Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:33:07.5675180Z Entering 'third_party/mimalloc' 2024-06-26T08:33:07.5712109Z Entering 'third_party/nccl/nccl' 2024-06-26T08:33:07.5748002Z Entering 'third_party/nlohmann' 2024-06-26T08:33:07.5784798Z Entering 'third_party/onnx' 2024-06-26T08:33:07.5834883Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:33:07.5869697Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:33:07.5908650Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:33:07.5947070Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:33:07.5982618Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:33:07.6019110Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:33:07.6054621Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:33:07.6090678Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:33:07.6125252Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:33:07.6161854Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:33:07.6196311Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:33:07.6230728Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:33:07.6268219Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:33:07.6316707Z Entering 'third_party/pocketfft' 2024-06-26T08:33:07.6355433Z Entering 'third_party/protobuf' 2024-06-26T08:33:07.6395289Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:33:07.6429624Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:33:07.6469282Z Entering 'third_party/psimd' 2024-06-26T08:33:07.6503579Z Entering 'third_party/pthreadpool' 2024-06-26T08:33:07.6540599Z Entering 'third_party/pybind11' 2024-06-26T08:33:07.6574098Z Entering 'third_party/python-peachpy' 2024-06-26T08:33:07.6607445Z Entering 'third_party/sleef' 2024-06-26T08:33:07.6644229Z Entering 'third_party/tensorpipe' 2024-06-26T08:33:07.6678901Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:33:07.6716807Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:33:07.6751044Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:33:07.6787518Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:33:07.6822442Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:33:07.6876494Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:33:07.6894668Z http.https://github.com/.extraheader 2024-06-26T08:33:07.6902598Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:33:07.6926106Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:33:07.7131895Z Entering 'android/libs/fbjni' 2024-06-26T08:33:07.7153809Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7177830Z Entering 'third_party/FP16' 2024-06-26T08:33:07.7200575Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7223362Z Entering 'third_party/FXdiv' 2024-06-26T08:33:07.7246343Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7270318Z Entering 'third_party/NNPACK' 2024-06-26T08:33:07.7293285Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7317013Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:33:07.7338853Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7362936Z Entering 'third_party/XNNPACK' 2024-06-26T08:33:07.7384323Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7418891Z Entering 'third_party/benchmark' 2024-06-26T08:33:07.7439731Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7463301Z Entering 'third_party/cpp-httplib' 2024-06-26T08:33:07.7485610Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7508464Z Entering 'third_party/cpuinfo' 2024-06-26T08:33:07.7531187Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7552859Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:33:07.7575473Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7599581Z Entering 'third_party/cutlass' 2024-06-26T08:33:07.7621199Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7648322Z Entering 'third_party/eigen' 2024-06-26T08:33:07.7670672Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7696488Z Entering 'third_party/fbgemm' 2024-06-26T08:33:07.7717913Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7742073Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:33:07.7764584Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7787527Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:33:07.7809441Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7834317Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:33:07.7856391Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7883663Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:33:07.7905021Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7929411Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:33:07.7950606Z http.https://github.com/.extraheader 2024-06-26T08:33:07.7975027Z Entering 'third_party/flatbuffers' 2024-06-26T08:33:07.7997541Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8023421Z Entering 'third_party/fmt' 2024-06-26T08:33:07.8047753Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8071079Z Entering 'third_party/foxi' 2024-06-26T08:33:07.8092819Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8116938Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:33:07.8139151Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8162814Z Entering 'third_party/gloo' 2024-06-26T08:33:07.8185081Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8209626Z Entering 'third_party/googletest' 2024-06-26T08:33:07.8230357Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8253109Z Entering 'third_party/ideep' 2024-06-26T08:33:07.8276512Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8299323Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:33:07.8321918Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8349608Z Entering 'third_party/ittapi' 2024-06-26T08:33:07.8374214Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8397817Z Entering 'third_party/kineto' 2024-06-26T08:33:07.8419172Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8440573Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:33:07.8462404Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8485390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:33:07.8506502Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8530410Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:33:07.8552339Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8575942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:33:07.8596029Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8621317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:33:07.8643860Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8668647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:33:07.8692027Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8718449Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:33:07.8739212Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8762612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:33:07.8784273Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8808000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:33:07.8829633Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8853374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:33:07.8875037Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8899351Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:33:07.8921609Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8945186Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:33:07.8966391Z http.https://github.com/.extraheader 2024-06-26T08:33:07.8988559Z Entering 'third_party/mimalloc' 2024-06-26T08:33:07.9011483Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9035074Z Entering 'third_party/nccl/nccl' 2024-06-26T08:33:07.9057103Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9079754Z Entering 'third_party/nlohmann' 2024-06-26T08:33:07.9101093Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9123251Z Entering 'third_party/onnx' 2024-06-26T08:33:07.9145019Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9182272Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:33:07.9204865Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9229367Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:33:07.9252285Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9278807Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:33:07.9300379Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9324681Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:33:07.9346583Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9371326Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:33:07.9392570Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9414960Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:33:07.9436859Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9459029Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:33:07.9480653Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9504995Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:33:07.9527034Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9548704Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:33:07.9572423Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9598043Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:33:07.9620219Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9643400Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:33:07.9664263Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9689403Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:33:07.9711456Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9735588Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:33:07.9757605Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9796951Z Entering 'third_party/pocketfft' 2024-06-26T08:33:07.9819666Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9843804Z Entering 'third_party/protobuf' 2024-06-26T08:33:07.9866497Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9894448Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:33:07.9915629Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9939414Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:33:07.9961416Z http.https://github.com/.extraheader 2024-06-26T08:33:07.9985559Z Entering 'third_party/psimd' 2024-06-26T08:33:08.0010347Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0033466Z Entering 'third_party/pthreadpool' 2024-06-26T08:33:08.0055648Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0081048Z Entering 'third_party/pybind11' 2024-06-26T08:33:08.0103431Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0127414Z Entering 'third_party/python-peachpy' 2024-06-26T08:33:08.0148936Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0170874Z Entering 'third_party/sleef' 2024-06-26T08:33:08.0190877Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0213064Z Entering 'third_party/tensorpipe' 2024-06-26T08:33:08.0235851Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0257783Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:33:08.0278855Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0303050Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:33:08.0325415Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0348667Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:33:08.0371265Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0393790Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:33:08.0415216Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0438014Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:33:08.0460114Z http.https://github.com/.extraheader 2024-06-26T08:33:08.0561706Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:33:08.0579088Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:33:08.0585704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:08.0586229Z ##[endgroup] 2024-06-26T08:33:08.8284460Z Cleaning up orphan processes