2024-06-26T07:01:25.2043124Z Current runner version: '2.317.0' 2024-06-26T07:01:25.2049385Z Runner name: 'i-02e64d1cff9e6c2ad' 2024-06-26T07:01:25.2050191Z Runner group name: 'Default' 2024-06-26T07:01:25.2051224Z Machine name: 'ip-10-0-19-247' 2024-06-26T07:01:25.2056167Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:01:25.2058155Z Actions: read 2024-06-26T07:01:25.2058731Z Attestations: read 2024-06-26T07:01:25.2059260Z Checks: read 2024-06-26T07:01:25.2059820Z Contents: read 2024-06-26T07:01:25.2060404Z Deployments: read 2024-06-26T07:01:25.2060942Z Discussions: read 2024-06-26T07:01:25.2061509Z Issues: read 2024-06-26T07:01:25.2062029Z Metadata: read 2024-06-26T07:01:25.2062574Z Packages: read 2024-06-26T07:01:25.2063092Z Pages: read 2024-06-26T07:01:25.2063649Z PullRequests: read 2024-06-26T07:01:25.2064429Z RepositoryProjects: read 2024-06-26T07:01:25.2065050Z SecurityEvents: read 2024-06-26T07:01:25.2065628Z Statuses: read 2024-06-26T07:01:25.2066174Z ##[endgroup] 2024-06-26T07:01:25.2069476Z Secret source: Actions 2024-06-26T07:01:25.2070335Z Prepare workflow directory 2024-06-26T07:01:25.6108304Z Prepare all required actions 2024-06-26T07:01:25.6292093Z Getting action download info 2024-06-26T07:01:25.8049074Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:01:26.1587071Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:01:28.9479978Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:01:29.0930801Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:01:29.3665666Z Getting action download info 2024-06-26T07:01:29.4594555Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:01:29.7020179Z Getting action download info 2024-06-26T07:01:29.8033866Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:01:29.9570191Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:01:29.9572459Z ##[group] Inputs 2024-06-26T07:01:29.9573018Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:01:29.9581709Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:01:29.9591411Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:01:29.9592714Z sync-tag: 2024-06-26T07:01:29.9593636Z timeout-minutes: 240 2024-06-26T07:01:29.9594049Z use-gha: 2024-06-26T07:01:29.9594406Z dashboard-tag: 2024-06-26T07:01:29.9594800Z s3-bucket: gha-artifacts 2024-06-26T07:01:29.9595237Z aws-role-to-assume: 2024-06-26T07:01:29.9595647Z ##[endgroup] 2024-06-26T07:01:29.9596644Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:01:30.0187747Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:01:30.0330494Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:01:30.0341328Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:30.0341948Z ##[endgroup] 2024-06-26T07:01:31.3924471Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:01:31.3925044Z with: 2024-06-26T07:01:31.3925766Z github-secret: *** 2024-06-26T07:01:31.3926828Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:01:31.3927845Z activate-with-label: false 2024-06-26T07:01:31.3928216Z label: with-ssh 2024-06-26T07:01:31.3928545Z remove-existing-keys: true 2024-06-26T07:01:31.3928909Z fail-silently: true 2024-06-26T07:01:31.3929231Z env: 2024-06-26T07:01:31.3929520Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:31.3929879Z ##[endgroup] 2024-06-26T07:01:31.4806433Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:01:31.4807463Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:01:31.7411741Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:01:31.8022031Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:01:31.8022811Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:01:31.8786668Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:01:31.8893316Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:01:31.8893872Z with: 2024-06-26T07:01:31.8894158Z submodules: recursive 2024-06-26T07:01:31.8894497Z fetch-depth: 0 2024-06-26T07:01:31.8894790Z env: 2024-06-26T07:01:31.8895069Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:31.8895417Z ##[endgroup] 2024-06-26T07:01:31.9083723Z ##[group]Run retry () { 2024-06-26T07:01:31.9084195Z retry () { 2024-06-26T07:01:31.9084777Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:01:31.9085356Z } 2024-06-26T07:01:31.9085726Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:01:31.9086286Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:01:31.9086834Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:31.9087346Z else 2024-06-26T07:01:31.9087768Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:31.9088252Z fi 2024-06-26T07:01:31.9088668Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:01:31.9096655Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:31.9097262Z env: 2024-06-26T07:01:31.9097598Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:31.9097977Z NO_SUDO: 2024-06-26T07:01:31.9098326Z ##[endgroup] 2024-06-26T07:01:31.9153411Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:34.2508390Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:01:34.2508834Z with: 2024-06-26T07:01:34.2509161Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:34.2509607Z fetch-depth: 0 2024-06-26T07:01:34.2509935Z submodules: recursive 2024-06-26T07:01:34.2510294Z quiet-checkout: true 2024-06-26T07:01:34.2510646Z repository: pytorch/pytorch 2024-06-26T07:01:34.2511143Z token: *** 2024-06-26T07:01:34.2511439Z ssh-strict: true 2024-06-26T07:01:34.2511786Z persist-credentials: true 2024-06-26T07:01:34.2512147Z clean: true 2024-06-26T07:01:34.2512499Z sparse-checkout-cone-mode: true 2024-06-26T07:01:34.2512898Z lfs: false 2024-06-26T07:01:34.2513198Z set-safe-directory: true 2024-06-26T07:01:34.2513551Z env: 2024-06-26T07:01:34.2513830Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:34.2514188Z ##[endgroup] 2024-06-26T07:01:34.3437310Z Syncing repository: pytorch/pytorch 2024-06-26T07:01:34.3438883Z ##[group]Getting Git version info 2024-06-26T07:01:34.3439646Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:34.3440510Z [command]/usr/bin/git version 2024-06-26T07:01:34.3440885Z git version 2.40.1 2024-06-26T07:01:34.3442209Z ##[endgroup] 2024-06-26T07:01:34.3453633Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c5fa3881-33ed-46b0-b569-841693670c3e' before making global git config changes 2024-06-26T07:01:34.3454901Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:01:34.3459533Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:34.3493999Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:34.3497789Z ##[group]Initializing the repository 2024-06-26T07:01:34.3500899Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:34.3520814Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:01:34.3521906Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:01:34.3522733Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:01:34.3523362Z hint: 2024-06-26T07:01:34.3524008Z hint: git config --global init.defaultBranch 2024-06-26T07:01:34.3524492Z hint: 2024-06-26T07:01:34.3525020Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:01:34.3525980Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:01:34.3526850Z hint: 2024-06-26T07:01:34.3527212Z hint: git branch -m 2024-06-26T07:01:34.3527970Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:01:34.3530552Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:01:34.3555116Z ##[endgroup] 2024-06-26T07:01:34.3555729Z ##[group]Disabling automatic garbage collection 2024-06-26T07:01:34.3557841Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:01:34.3582023Z ##[endgroup] 2024-06-26T07:01:34.3583031Z ##[group]Setting up auth 2024-06-26T07:01:34.3589596Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:01:34.3611901Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:01:34.3810643Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:01:34.3836926Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:01:34.4039334Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:01:34.4083680Z ##[endgroup] 2024-06-26T07:01:34.4084368Z ##[group]Fetching the repository 2024-06-26T07:01:34.4089720Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:01:36.4027423Z remote: Enumerating objects: 987392 2024-06-26T07:01:36.4028675Z remote: Enumerating objects: 990209, done. 2024-06-26T07:01:36.4032046Z remote: Counting objects: 0% (1/2817) 2024-06-26T07:01:36.4033234Z remote: Counting objects: 1% (29/2817) 2024-06-26T07:01:36.4034317Z remote: Counting objects: 2% (57/2817) 2024-06-26T07:01:36.4034951Z remote: Counting objects: 3% (85/2817) 2024-06-26T07:01:36.4035548Z remote: Counting objects: 4% (113/2817) 2024-06-26T07:01:36.4036156Z remote: Counting objects: 5% (141/2817) 2024-06-26T07:01:36.4038634Z remote: Counting objects: 6% (170/2817) 2024-06-26T07:01:36.4039262Z remote: Counting objects: 7% (198/2817) 2024-06-26T07:01:36.4040958Z remote: Counting objects: 8% (226/2817) 2024-06-26T07:01:36.4041572Z remote: Counting objects: 9% (254/2817) 2024-06-26T07:01:36.4042162Z remote: Counting objects: 10% (282/2817) 2024-06-26T07:01:36.4043648Z remote: Counting objects: 11% (310/2817) 2024-06-26T07:01:36.4044265Z remote: Counting objects: 12% (339/2817) 2024-06-26T07:01:36.4044868Z remote: Counting objects: 13% (367/2817) 2024-06-26T07:01:36.4045456Z remote: Counting objects: 14% (395/2817) 2024-06-26T07:01:36.4046133Z remote: Counting objects: 15% (423/2817) 2024-06-26T07:01:36.4046728Z remote: Counting objects: 16% (451/2817) 2024-06-26T07:01:36.4047322Z remote: Counting objects: 17% (479/2817) 2024-06-26T07:01:36.4047911Z remote: Counting objects: 18% (508/2817) 2024-06-26T07:01:36.4048676Z remote: Counting objects: 19% (536/2817) 2024-06-26T07:01:36.4049401Z remote: Counting objects: 20% (564/2817) 2024-06-26T07:01:36.4050025Z remote: Counting objects: 21% (592/2817) 2024-06-26T07:01:36.4050666Z remote: Counting objects: 22% (620/2817) 2024-06-26T07:01:36.4051289Z remote: Counting objects: 23% (648/2817) 2024-06-26T07:01:36.4051929Z remote: Counting objects: 24% (677/2817) 2024-06-26T07:01:36.4052565Z remote: Counting objects: 25% (705/2817) 2024-06-26T07:01:36.4053157Z remote: Counting objects: 26% (733/2817) 2024-06-26T07:01:36.4053756Z remote: Counting objects: 27% (761/2817) 2024-06-26T07:01:36.4054610Z remote: Counting objects: 28% (789/2817) 2024-06-26T07:01:36.4055216Z remote: Counting objects: 29% (817/2817) 2024-06-26T07:01:36.4055808Z remote: Counting objects: 30% (846/2817) 2024-06-26T07:01:36.4056486Z remote: Counting objects: 31% (874/2817) 2024-06-26T07:01:36.4057098Z remote: Counting objects: 32% (902/2817) 2024-06-26T07:01:36.4057703Z remote: Counting objects: 33% (930/2817) 2024-06-26T07:01:36.4058297Z remote: Counting objects: 34% (958/2817) 2024-06-26T07:01:36.4059068Z remote: Counting objects: 35% (986/2817) 2024-06-26T07:01:36.4059722Z remote: Counting objects: 36% (1015/2817) 2024-06-26T07:01:36.4060343Z remote: Counting objects: 37% (1043/2817) 2024-06-26T07:01:36.4060992Z remote: Counting objects: 38% (1071/2817) 2024-06-26T07:01:36.4061804Z remote: Counting objects: 39% (1099/2817) 2024-06-26T07:01:36.4062445Z remote: Counting objects: 40% (1127/2817) 2024-06-26T07:01:36.4063060Z remote: Counting objects: 41% (1155/2817) 2024-06-26T07:01:36.4064029Z remote: Counting objects: 42% (1184/2817) 2024-06-26T07:01:36.4064757Z remote: Counting objects: 43% (1212/2817) 2024-06-26T07:01:36.4065373Z remote: Counting objects: 44% (1240/2817) 2024-06-26T07:01:36.4066078Z remote: Counting objects: 45% (1268/2817) 2024-06-26T07:01:36.4067047Z remote: Counting objects: 46% (1296/2817) 2024-06-26T07:01:36.4067659Z remote: Counting objects: 47% (1324/2817) 2024-06-26T07:01:36.4068295Z remote: Counting objects: 48% (1353/2817) 2024-06-26T07:01:36.4069105Z remote: Counting objects: 49% (1381/2817) 2024-06-26T07:01:36.4069747Z remote: Counting objects: 50% (1409/2817) 2024-06-26T07:01:36.4070353Z remote: Counting objects: 51% (1437/2817) 2024-06-26T07:01:36.4071025Z remote: Counting objects: 52% (1465/2817) 2024-06-26T07:01:36.4071811Z remote: Counting objects: 53% (1494/2817) 2024-06-26T07:01:36.4072423Z remote: Counting objects: 54% (1522/2817) 2024-06-26T07:01:36.4073036Z remote: Counting objects: 55% (1550/2817) 2024-06-26T07:01:36.4073709Z remote: Counting objects: 56% (1578/2817) 2024-06-26T07:01:36.4074519Z remote: Counting objects: 57% (1606/2817) 2024-06-26T07:01:36.4075272Z remote: Counting objects: 58% (1634/2817) 2024-06-26T07:01:36.4075973Z remote: Counting objects: 59% (1663/2817) 2024-06-26T07:01:36.4076725Z remote: Counting objects: 60% (1691/2817) 2024-06-26T07:01:36.4077418Z remote: Counting objects: 61% (1719/2817) 2024-06-26T07:01:36.4078018Z remote: Counting objects: 62% (1747/2817) 2024-06-26T07:01:36.4078742Z remote: Counting objects: 63% (1775/2817) 2024-06-26T07:01:36.4079468Z remote: Counting objects: 64% (1803/2817) 2024-06-26T07:01:36.4080085Z remote: Counting objects: 65% (1832/2817) 2024-06-26T07:01:36.4080778Z remote: Counting objects: 66% (1860/2817) 2024-06-26T07:01:36.4081521Z remote: Counting objects: 67% (1888/2817) 2024-06-26T07:01:36.4082140Z remote: Counting objects: 68% (1916/2817) 2024-06-26T07:01:36.4082756Z remote: Counting objects: 69% (1944/2817) 2024-06-26T07:01:36.4083378Z remote: Counting objects: 70% (1972/2817) 2024-06-26T07:01:36.4084007Z remote: Counting objects: 71% (2001/2817) 2024-06-26T07:01:36.4084617Z remote: Counting objects: 72% (2029/2817) 2024-06-26T07:01:36.4085244Z remote: Counting objects: 73% (2057/2817) 2024-06-26T07:01:36.4085915Z remote: Counting objects: 74% (2085/2817) 2024-06-26T07:01:36.4086704Z remote: Counting objects: 75% (2113/2817) 2024-06-26T07:01:36.4087393Z remote: Counting objects: 76% (2141/2817) 2024-06-26T07:01:36.4088006Z remote: Counting objects: 77% (2170/2817) 2024-06-26T07:01:36.4088617Z remote: Counting objects: 78% (2198/2817) 2024-06-26T07:01:36.4089344Z remote: Counting objects: 79% (2226/2817) 2024-06-26T07:01:36.4089967Z remote: Counting objects: 80% (2254/2817) 2024-06-26T07:01:36.4090588Z remote: Counting objects: 81% (2282/2817) 2024-06-26T07:01:36.4091214Z remote: Counting objects: 82% (2310/2817) 2024-06-26T07:01:36.4091830Z remote: Counting objects: 83% (2339/2817) 2024-06-26T07:01:36.4092451Z remote: Counting objects: 84% (2367/2817) 2024-06-26T07:01:36.4093258Z remote: Counting objects: 85% (2395/2817) 2024-06-26T07:01:36.4093982Z remote: Counting objects: 86% (2423/2817) 2024-06-26T07:01:36.4094591Z remote: Counting objects: 87% (2451/2817) 2024-06-26T07:01:36.4095200Z remote: Counting objects: 88% (2479/2817) 2024-06-26T07:01:36.4095816Z remote: Counting objects: 89% (2508/2817) 2024-06-26T07:01:36.4096431Z remote: Counting objects: 90% (2536/2817) 2024-06-26T07:01:36.4097067Z remote: Counting objects: 91% (2564/2817) 2024-06-26T07:01:36.4097685Z remote: Counting objects: 92% (2592/2817) 2024-06-26T07:01:36.4098302Z remote: Counting objects: 93% (2620/2817) 2024-06-26T07:01:36.4098910Z remote: Counting objects: 94% (2648/2817) 2024-06-26T07:01:36.4099530Z remote: Counting objects: 95% (2677/2817) 2024-06-26T07:01:36.4100139Z remote: Counting objects: 96% (2705/2817) 2024-06-26T07:01:36.4100830Z remote: Counting objects: 97% (2733/2817) 2024-06-26T07:01:36.4101459Z remote: Counting objects: 98% (2761/2817) 2024-06-26T07:01:36.4102080Z remote: Counting objects: 99% (2789/2817) 2024-06-26T07:01:36.4102698Z remote: Counting objects: 100% (2817/2817) 2024-06-26T07:01:36.4103357Z remote: Counting objects: 100% (2817/2817), done. 2024-06-26T07:01:36.4250819Z remote: Compressing objects: 0% (1/1140) 2024-06-26T07:01:36.4788876Z remote: Compressing objects: 1% (12/1140) 2024-06-26T07:01:36.5205054Z remote: Compressing objects: 2% (23/1140) 2024-06-26T07:01:36.5509410Z remote: Compressing objects: 3% (35/1140) 2024-06-26T07:01:36.5696470Z remote: Compressing objects: 4% (46/1140) 2024-06-26T07:01:36.5968324Z remote: Compressing objects: 5% (57/1140) 2024-06-26T07:01:36.6175793Z remote: Compressing objects: 6% (69/1140) 2024-06-26T07:01:36.7081100Z remote: Compressing objects: 7% (80/1140) 2024-06-26T07:01:36.7676788Z remote: Compressing objects: 8% (92/1140) 2024-06-26T07:01:36.8121295Z remote: Compressing objects: 9% (103/1140) 2024-06-26T07:01:36.8286147Z remote: Compressing objects: 10% (114/1140) 2024-06-26T07:01:36.8377591Z remote: Compressing objects: 11% (126/1140) 2024-06-26T07:01:36.8445646Z remote: Compressing objects: 12% (137/1140) 2024-06-26T07:01:36.8489156Z remote: Compressing objects: 13% (149/1140) 2024-06-26T07:01:36.8489800Z remote: Compressing objects: 14% (160/1140) 2024-06-26T07:01:36.8552459Z remote: Compressing objects: 15% (171/1140) 2024-06-26T07:01:36.8553107Z remote: Compressing objects: 16% (183/1140) 2024-06-26T07:01:36.8561333Z remote: Compressing objects: 17% (194/1140) 2024-06-26T07:01:36.8576313Z remote: Compressing objects: 18% (206/1140) 2024-06-26T07:01:36.8594862Z remote: Compressing objects: 19% (217/1140) 2024-06-26T07:01:36.8616174Z remote: Compressing objects: 20% (228/1140) 2024-06-26T07:01:36.8634870Z remote: Compressing objects: 21% (240/1140) 2024-06-26T07:01:36.8662052Z remote: Compressing objects: 22% (251/1140) 2024-06-26T07:01:36.8666297Z remote: Compressing objects: 23% (263/1140) 2024-06-26T07:01:36.8673444Z remote: Compressing objects: 24% (274/1140) 2024-06-26T07:01:36.8683577Z remote: Compressing objects: 25% (285/1140) 2024-06-26T07:01:36.8697042Z remote: Compressing objects: 26% (297/1140) 2024-06-26T07:01:36.8700880Z remote: Compressing objects: 27% (308/1140) 2024-06-26T07:01:36.8707285Z remote: Compressing objects: 28% (320/1140) 2024-06-26T07:01:36.8716017Z remote: Compressing objects: 29% (331/1140) 2024-06-26T07:01:36.8725978Z remote: Compressing objects: 30% (342/1140) 2024-06-26T07:01:36.8730487Z remote: Compressing objects: 31% (354/1140) 2024-06-26T07:01:36.8738436Z remote: Compressing objects: 32% (365/1140) 2024-06-26T07:01:36.8750487Z remote: Compressing objects: 33% (377/1140) 2024-06-26T07:01:36.8763531Z remote: Compressing objects: 34% (388/1140) 2024-06-26T07:01:36.8778105Z remote: Compressing objects: 35% (399/1140) 2024-06-26T07:01:36.8798080Z remote: Compressing objects: 36% (411/1140) 2024-06-26T07:01:36.8824023Z remote: Compressing objects: 37% (422/1140) 2024-06-26T07:01:36.8833870Z remote: Compressing objects: 38% (434/1140) 2024-06-26T07:01:36.8840679Z remote: Compressing objects: 39% (445/1140) 2024-06-26T07:01:36.8859998Z remote: Compressing objects: 40% (456/1140) 2024-06-26T07:01:36.8878162Z remote: Compressing objects: 41% (468/1140) 2024-06-26T07:01:36.8901735Z remote: Compressing objects: 42% (479/1140) 2024-06-26T07:01:36.8910891Z remote: Compressing objects: 43% (491/1140) 2024-06-26T07:01:36.8916100Z remote: Compressing objects: 44% (502/1140) 2024-06-26T07:01:36.8920147Z remote: Compressing objects: 45% (513/1140) 2024-06-26T07:01:36.8929940Z remote: Compressing objects: 46% (525/1140) 2024-06-26T07:01:36.8944951Z remote: Compressing objects: 47% (536/1140) 2024-06-26T07:01:36.8950849Z remote: Compressing objects: 48% (548/1140) 2024-06-26T07:01:36.8954473Z remote: Compressing objects: 49% (559/1140) 2024-06-26T07:01:36.8960505Z remote: Compressing objects: 50% (570/1140) 2024-06-26T07:01:36.8975310Z remote: Compressing objects: 51% (582/1140) 2024-06-26T07:01:36.8996812Z remote: Compressing objects: 52% (593/1140) 2024-06-26T07:01:36.9015839Z remote: Compressing objects: 53% (605/1140) 2024-06-26T07:01:36.9030290Z remote: Compressing objects: 54% (616/1140) 2024-06-26T07:01:36.9039564Z remote: Compressing objects: 55% (627/1140) 2024-06-26T07:01:36.9045777Z remote: Compressing objects: 56% (639/1140) 2024-06-26T07:01:36.9056185Z remote: Compressing objects: 57% (650/1140) 2024-06-26T07:01:36.9067248Z remote: Compressing objects: 58% (662/1140) 2024-06-26T07:01:36.9078499Z remote: Compressing objects: 59% (673/1140) 2024-06-26T07:01:36.9089664Z remote: Compressing objects: 60% (684/1140) 2024-06-26T07:01:36.9097288Z remote: Compressing objects: 61% (696/1140) 2024-06-26T07:01:36.9110494Z remote: Compressing objects: 62% (707/1140) 2024-06-26T07:01:36.9121132Z remote: Compressing objects: 63% (719/1140) 2024-06-26T07:01:36.9130322Z remote: Compressing objects: 64% (730/1140) 2024-06-26T07:01:36.9139643Z remote: Compressing objects: 65% (741/1140) 2024-06-26T07:01:36.9147664Z remote: Compressing objects: 66% (753/1140) 2024-06-26T07:01:36.9154634Z remote: Compressing objects: 67% (764/1140) 2024-06-26T07:01:36.9158356Z remote: Compressing objects: 68% (776/1140) 2024-06-26T07:01:36.9165564Z remote: Compressing objects: 69% (787/1140) 2024-06-26T07:01:36.9171710Z remote: Compressing objects: 70% (798/1140) 2024-06-26T07:01:36.9174518Z remote: Compressing objects: 71% (810/1140) 2024-06-26T07:01:36.9179344Z remote: Compressing objects: 72% (821/1140) 2024-06-26T07:01:36.9179990Z remote: Compressing objects: 73% (833/1140) 2024-06-26T07:01:36.9181908Z remote: Compressing objects: 74% (844/1140) 2024-06-26T07:01:36.9182549Z remote: Compressing objects: 75% (855/1140) 2024-06-26T07:01:36.9184655Z remote: Compressing objects: 76% (867/1140) 2024-06-26T07:01:36.9185499Z remote: Compressing objects: 77% (878/1140) 2024-06-26T07:01:36.9186146Z remote: Compressing objects: 78% (890/1140) 2024-06-26T07:01:36.9186783Z remote: Compressing objects: 79% (901/1140) 2024-06-26T07:01:36.9190135Z remote: Compressing objects: 80% (912/1140) 2024-06-26T07:01:36.9192383Z remote: Compressing objects: 81% (924/1140) 2024-06-26T07:01:36.9198863Z remote: Compressing objects: 82% (935/1140) 2024-06-26T07:01:36.9205838Z remote: Compressing objects: 83% (947/1140) 2024-06-26T07:01:36.9208499Z remote: Compressing objects: 84% (958/1140) 2024-06-26T07:01:36.9209285Z remote: Compressing objects: 85% (969/1140) 2024-06-26T07:01:36.9210016Z remote: Compressing objects: 86% (981/1140) 2024-06-26T07:01:36.9210662Z remote: Compressing objects: 87% (992/1140) 2024-06-26T07:01:36.9211509Z remote: Compressing objects: 88% (1004/1140) 2024-06-26T07:01:36.9212239Z remote: Compressing objects: 89% (1015/1140) 2024-06-26T07:01:36.9212891Z remote: Compressing objects: 90% (1026/1140) 2024-06-26T07:01:36.9213542Z remote: Compressing objects: 91% (1038/1140) 2024-06-26T07:01:36.9214299Z remote: Compressing objects: 92% (1049/1140) 2024-06-26T07:01:36.9215077Z remote: Compressing objects: 93% (1061/1140) 2024-06-26T07:01:36.9215724Z remote: Compressing objects: 94% (1072/1140) 2024-06-26T07:01:36.9216498Z remote: Compressing objects: 95% (1083/1140) 2024-06-26T07:01:36.9217135Z remote: Compressing objects: 96% (1095/1140) 2024-06-26T07:01:36.9217777Z remote: Compressing objects: 97% (1106/1140) 2024-06-26T07:01:36.9218405Z remote: Compressing objects: 98% (1118/1140) 2024-06-26T07:01:36.9223188Z remote: Compressing objects: 99% (1129/1140) 2024-06-26T07:01:36.9224153Z remote: Compressing objects: 100% (1140/1140) 2024-06-26T07:01:36.9224836Z remote: Compressing objects: 100% (1140/1140), done. 2024-06-26T07:01:56.4977367Z remote: Total 990209 (delta 2280), reused 2016 (delta 1676), pack-reused 987392 2024-06-26T07:02:22.2954575Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:02:22.2974019Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:22.2978174Z ##[endgroup] 2024-06-26T07:02:22.2978963Z ##[group]Determining the checkout info 2024-06-26T07:02:22.2979924Z ##[endgroup] 2024-06-26T07:02:22.2980640Z ##[group]Checking out the ref 2024-06-26T07:02:22.2982968Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:23.5222161Z ##[endgroup] 2024-06-26T07:02:23.5222916Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:02:23.5227004Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:02:23.5269992Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:02:23.5291206Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:02:23.5312375Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:02:23.5329831Z ##[endgroup] 2024-06-26T07:02:23.5330612Z ##[group]Fetching submodules 2024-06-26T07:02:23.5334242Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:02:23.5546936Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:02:23.5752427Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:02:23.5755600Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:02:23.5757385Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:02:23.5759032Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:02:23.5760832Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:23.5762601Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:02:23.5763988Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:02:23.5765432Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:02:23.5767087Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:02:23.5768497Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:02:23.5770059Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:02:23.5772054Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:02:23.5774605Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:02:23.5777318Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:02:23.5779698Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:02:23.5782284Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:02:23.5786194Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:23.5788847Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:02:23.5791693Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:02:23.5794487Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:02:23.5797590Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:02:23.5800748Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:02:23.5803975Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:02:23.5807201Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:02:23.5810471Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:02:23.5813631Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:02:23.5817267Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:02:23.5820448Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:02:23.5824117Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:02:23.5827995Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:02:23.5831800Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:02:23.5835225Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:02:23.5839000Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:02:23.5842661Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:02:23.5846722Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:02:23.5865468Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:02:23.8435685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:02:24.0001725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:02:24.1583643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:02:24.4044195Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:02:26.5244970Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:02:36.3382132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:02:36.7293874Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:02:37.2266122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:02:37.8460419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:02:39.0697263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:02:40.8105672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:02:46.7937794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:02:48.3354643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:02:49.9885280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:02:51.2516702Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:02:51.4322063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:02:51.7973440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:02:52.1111505Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:02:53.1526885Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:02:53.5401800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:02:53.8129582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:02:55.3779274Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:02:56.1432123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:02:56.9265869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:03:02.8469459Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:03:05.2137270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:03:10.7936439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:03:11.0475412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:03:19.3618188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:03:19.5189909Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:03:19.6903368Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:03:20.7869374Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:03:21.0538538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:03:21.6652808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:03:22.0701227Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:03:22.0784524Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:03:22.0844103Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:03:22.1026727Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:03:22.1331343Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:03:22.9018363Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:03:22.9190796Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:03:22.9542817Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:03:23.0387207Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:03:23.0657633Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:03:23.4942753Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:03:23.7075866Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:03:23.7718134Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:03:23.7728977Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:23.7732340Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:23.7734343Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:23.7736064Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:23.7737820Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:23.7752280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:03:24.8015966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:03:25.4166873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:03:27.1533013Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:03:28.1176602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:03:28.5166777Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:03:28.6006467Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:03:28.9446415Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:03:28.9998093Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:03:29.0082582Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:03:29.0995618Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:03:29.1277131Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:03:29.1341742Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:03:29.1655006Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:03:29.1852226Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:03:29.2218831Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:03:29.2315094Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:03:29.2325545Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:29.2341725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:03:41.3823614Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:03:41.3960819Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:03:41.4739649Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:03:41.4752067Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:41.4753818Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:41.4755926Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:41.4771134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:03:41.9854292Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:03:43.2019791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:03:44.2533413Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:03:44.2543542Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:44.2545846Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:44.2547901Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:44.2549872Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:44.2551906Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:44.2554334Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:44.2556635Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:44.2558491Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:44.2576568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:03:45.3253397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:03:45.6859103Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:03:46.8970013Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:03:47.1700618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:03:47.6209856Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:03:48.5856259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:03:54.4375706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:03:54.7990225Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:03:54.8126988Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:03:54.8415390Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:03:54.8509228Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:03:54.8518800Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:54.8535323Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:03:55.1919179Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:03:55.2052992Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:03:55.2390069Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:03:55.3192919Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:03:55.3311100Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:03:55.3654256Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:03:55.4161497Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:03:55.4467198Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:03:55.4666367Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:03:55.5524679Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:03:55.8242673Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:03:55.8265243Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:55.8267288Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:55.8283679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:03:56.2806245Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:03:57.4077895Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:03:57.4357118Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:03:57.4884779Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:03:57.4897165Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:57.4903985Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:57.4905855Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:57.4907473Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:57.4909565Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:57.4911592Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:57.4913759Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:57.4915355Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:57.4927879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:03:57.9621380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:03:58.9732133Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:03:59.2808591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:04:05.1115082Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:04:05.3862107Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:04:05.5678044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:04:05.8341451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:04:13.3262933Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:04:13.3594250Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:04:13.3716651Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:04:13.4569732Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:04:13.4662118Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:04:13.4768465Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:04:13.4881297Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:04:13.4891563Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:13.4893554Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:13.4909713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:04:15.3059604Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:04:16.5022416Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:04:16.5389629Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:04:16.8987955Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:04:16.9073944Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:04:17.1268821Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:04:17.1282427Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:17.1284447Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:17.1299619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:04:17.5831653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:04:18.5652207Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:04:18.6277913Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:04:18.6338461Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:04:18.6426978Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:04:18.6708238Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:04:18.6932297Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:04:18.7262692Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:04:18.7467256Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:04:18.7478319Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:18.7479911Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:18.7481569Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:18.7483320Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:18.7502188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:04:19.7195424Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:04:19.9582331Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:04:22.0077421Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:04:23.0848163Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:04:23.0960747Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:04:23.1526895Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:04:23.1753113Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:04:23.1763811Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:23.1779663Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:04:23.3837254Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:04:23.3861702Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:04:23.4059038Z Entering 'android/libs/fbjni' 2024-06-26T07:04:23.4085298Z Entering 'third_party/FP16' 2024-06-26T07:04:23.4111393Z Entering 'third_party/FXdiv' 2024-06-26T07:04:23.4137132Z Entering 'third_party/NNPACK' 2024-06-26T07:04:23.4165855Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:23.4191043Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:23.4230990Z Entering 'third_party/benchmark' 2024-06-26T07:04:23.4256821Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:23.4283921Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:23.4311844Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:23.4337565Z Entering 'third_party/cutlass' 2024-06-26T07:04:23.4368489Z Entering 'third_party/eigen' 2024-06-26T07:04:23.4402639Z Entering 'third_party/fbgemm' 2024-06-26T07:04:23.4429507Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:23.4457730Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:23.4484553Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:23.4517449Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:23.4544340Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:23.4573495Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:23.4601761Z Entering 'third_party/fmt' 2024-06-26T07:04:23.4630439Z Entering 'third_party/foxi' 2024-06-26T07:04:23.4656752Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:23.4683930Z Entering 'third_party/gloo' 2024-06-26T07:04:23.4712156Z Entering 'third_party/googletest' 2024-06-26T07:04:23.4738672Z Entering 'third_party/ideep' 2024-06-26T07:04:23.4766821Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:23.4800125Z Entering 'third_party/ittapi' 2024-06-26T07:04:23.4830047Z Entering 'third_party/kineto' 2024-06-26T07:04:23.4859498Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:23.4888835Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:23.4917531Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:23.4946063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:23.4972522Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:23.5000288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:23.5030023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:23.5055433Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:23.5083596Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:23.5111169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:23.5137723Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:23.5167300Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:23.5194014Z Entering 'third_party/mimalloc' 2024-06-26T07:04:23.5220656Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:23.5251134Z Entering 'third_party/nlohmann' 2024-06-26T07:04:23.5278575Z Entering 'third_party/onnx' 2024-06-26T07:04:23.5316685Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:23.5346246Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:23.5375874Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:23.5404265Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:23.5431834Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:23.5459266Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:23.5486926Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:23.5515613Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:23.5541876Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:23.5570515Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:23.5595646Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:23.5624419Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:23.5653120Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:23.5695930Z Entering 'third_party/pocketfft' 2024-06-26T07:04:23.5725810Z Entering 'third_party/protobuf' 2024-06-26T07:04:23.5755764Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:23.5783473Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:23.5812418Z Entering 'third_party/psimd' 2024-06-26T07:04:23.5839108Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:23.5866545Z Entering 'third_party/pybind11' 2024-06-26T07:04:23.5895848Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:23.5924551Z Entering 'third_party/sleef' 2024-06-26T07:04:23.5953981Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:23.5980231Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:23.6008586Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:23.6035958Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:23.6062882Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:23.6090281Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:23.6130722Z ##[endgroup] 2024-06-26T07:04:23.6132991Z ##[group]Persisting credentials for submodules 2024-06-26T07:04:23.6136394Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:04:23.6341642Z Entering 'android/libs/fbjni' 2024-06-26T07:04:23.6377864Z Entering 'third_party/FP16' 2024-06-26T07:04:23.6413700Z Entering 'third_party/FXdiv' 2024-06-26T07:04:23.6449421Z Entering 'third_party/NNPACK' 2024-06-26T07:04:23.6483082Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:23.6518767Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:23.6564333Z Entering 'third_party/benchmark' 2024-06-26T07:04:23.6602299Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:23.6637186Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:23.6672454Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:23.6709428Z Entering 'third_party/cutlass' 2024-06-26T07:04:23.6749708Z Entering 'third_party/eigen' 2024-06-26T07:04:23.6786468Z Entering 'third_party/fbgemm' 2024-06-26T07:04:23.6820852Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:23.6855249Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:23.6890659Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:23.6930694Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:23.6965294Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:23.7000525Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:23.7037765Z Entering 'third_party/fmt' 2024-06-26T07:04:23.7071875Z Entering 'third_party/foxi' 2024-06-26T07:04:23.7108532Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:23.7144458Z Entering 'third_party/gloo' 2024-06-26T07:04:23.7179576Z Entering 'third_party/googletest' 2024-06-26T07:04:23.7216160Z Entering 'third_party/ideep' 2024-06-26T07:04:23.7251237Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:23.7288779Z Entering 'third_party/ittapi' 2024-06-26T07:04:23.7325319Z Entering 'third_party/kineto' 2024-06-26T07:04:23.7362277Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:23.7396175Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:23.7430666Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:23.7465298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:23.7499265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:23.7534498Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:23.7571143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:23.7605694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:23.7638705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:23.7674282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:23.7709118Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:23.7741668Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:23.7777431Z Entering 'third_party/mimalloc' 2024-06-26T07:04:23.7812975Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:23.7847863Z Entering 'third_party/nlohmann' 2024-06-26T07:04:23.7884041Z Entering 'third_party/onnx' 2024-06-26T07:04:23.7932236Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:23.7967190Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:23.8005275Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:23.8040387Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:23.8075285Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:23.8107903Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:23.8141712Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:23.8176436Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:23.8210652Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:23.8242653Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:23.8273939Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:23.8309543Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:23.8343668Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:23.8393585Z Entering 'third_party/pocketfft' 2024-06-26T07:04:23.8429826Z Entering 'third_party/protobuf' 2024-06-26T07:04:23.8467160Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:23.8499543Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:23.8536441Z Entering 'third_party/psimd' 2024-06-26T07:04:23.8572468Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:23.8609660Z Entering 'third_party/pybind11' 2024-06-26T07:04:23.8645888Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:23.8681563Z Entering 'third_party/sleef' 2024-06-26T07:04:23.8714998Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:23.8747588Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:23.8780848Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:23.8815288Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:23.8851077Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:23.8882251Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:23.8929876Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:04:23.9141880Z Entering 'android/libs/fbjni' 2024-06-26T07:04:23.9171426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:04:23.9182932Z Entering 'third_party/FP16' 2024-06-26T07:04:23.9215126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:04:23.9227391Z Entering 'third_party/FXdiv' 2024-06-26T07:04:23.9257751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:04:23.9271785Z Entering 'third_party/NNPACK' 2024-06-26T07:04:23.9305228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:04:23.9318370Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:23.9348860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:04:23.9360998Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:23.9393330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:04:23.9416975Z Entering 'third_party/benchmark' 2024-06-26T07:04:23.9449786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:23.9462060Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:23.9493397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:04:23.9506599Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:23.9537450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:23.9549879Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:23.9580946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:04:23.9593114Z Entering 'third_party/cutlass' 2024-06-26T07:04:23.9624611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:23.9643364Z Entering 'third_party/eigen' 2024-06-26T07:04:23.9675360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:04:23.9689946Z Entering 'third_party/fbgemm' 2024-06-26T07:04:23.9721329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:04:23.9735015Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:23.9765250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:04:23.9776449Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:23.9806811Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:23.9820241Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:23.9851380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:23.9866868Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:23.9897683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:23.9910043Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:23.9942081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:04:23.9956829Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:23.9989871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:04:24.0005417Z Entering 'third_party/fmt' 2024-06-26T07:04:24.0036795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:24.0051324Z Entering 'third_party/foxi' 2024-06-26T07:04:24.0082522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:04:24.0096332Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:24.0127245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:04:24.0139472Z Entering 'third_party/gloo' 2024-06-26T07:04:24.0171097Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:04:24.0183723Z Entering 'third_party/googletest' 2024-06-26T07:04:24.0215903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:24.0229282Z Entering 'third_party/ideep' 2024-06-26T07:04:24.0260699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:04:24.0273788Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:24.0307513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:04:24.0325188Z Entering 'third_party/ittapi' 2024-06-26T07:04:24.0358808Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:04:24.0371942Z Entering 'third_party/kineto' 2024-06-26T07:04:24.0404240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:04:24.0417597Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:24.0450570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:04:24.0462570Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:24.0494346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:04:24.0509658Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:24.0541175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:04:24.0554219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:24.0586056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:24.0598382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:24.0628604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:04:24.0639823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:24.0671903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:04:24.0686596Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:24.0717981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:04:24.0730540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:24.0764652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:24.0777521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:24.0809854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:04:24.0823004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:24.0854593Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:04:24.0868070Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:24.0899896Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:04:24.0912793Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:24.0944245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:04:24.0958118Z Entering 'third_party/mimalloc' 2024-06-26T07:04:24.0991956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:04:24.1004521Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:24.1036651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:04:24.1050760Z Entering 'third_party/nlohmann' 2024-06-26T07:04:24.1081620Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:04:24.1093861Z Entering 'third_party/onnx' 2024-06-26T07:04:24.1125128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:04:24.1149571Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:24.1180608Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:24.1192929Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:24.1225488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:24.1240184Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:24.1271834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:04:24.1286077Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:24.1317846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:24.1330999Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:24.1363771Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:24.1375953Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:24.1408213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:04:24.1420272Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:24.1452734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:04:24.1466057Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:24.1497502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:04:24.1510132Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:24.1541606Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:04:24.1553535Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:24.1585266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:04:24.1596157Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:24.1626789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:04:24.1639655Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:24.1670566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:04:24.1684989Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:24.1715691Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:04:24.1744824Z Entering 'third_party/pocketfft' 2024-06-26T07:04:24.1777396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:04:24.1790534Z Entering 'third_party/protobuf' 2024-06-26T07:04:24.1821042Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:04:24.1836190Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:24.1867857Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:24.1879729Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:24.1910848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:24.1925472Z Entering 'third_party/psimd' 2024-06-26T07:04:24.1956121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:04:24.1969378Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:24.2002932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:04:24.2015650Z Entering 'third_party/pybind11' 2024-06-26T07:04:24.2047314Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:24.2059168Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:24.2091244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:04:24.2102692Z Entering 'third_party/sleef' 2024-06-26T07:04:24.2133020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:04:24.2147141Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:24.2176937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:04:24.2188929Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:24.2220377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:24.2233128Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:24.2265272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:04:24.2277762Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:24.2307984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:04:24.2319699Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:24.2351031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:24.2363258Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:24.2395989Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:04:24.3637886Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:04:24.3846985Z Entering 'android/libs/fbjni' 2024-06-26T07:04:24.3875220Z Entering 'third_party/FP16' 2024-06-26T07:04:24.3902828Z Entering 'third_party/FXdiv' 2024-06-26T07:04:24.3931736Z Entering 'third_party/NNPACK' 2024-06-26T07:04:24.3961315Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:24.3988230Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:24.4025718Z Entering 'third_party/benchmark' 2024-06-26T07:04:24.4053724Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:24.4083132Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:24.4112512Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:24.4141737Z Entering 'third_party/cutlass' 2024-06-26T07:04:24.4175681Z Entering 'third_party/eigen' 2024-06-26T07:04:24.4204807Z Entering 'third_party/fbgemm' 2024-06-26T07:04:24.4232457Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:24.4261713Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:24.4289459Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:24.4320898Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:24.4349185Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:24.4378459Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:24.4409858Z Entering 'third_party/fmt' 2024-06-26T07:04:24.4437322Z Entering 'third_party/foxi' 2024-06-26T07:04:24.4465156Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:24.4493380Z Entering 'third_party/gloo' 2024-06-26T07:04:24.4521128Z Entering 'third_party/googletest' 2024-06-26T07:04:24.4549146Z Entering 'third_party/ideep' 2024-06-26T07:04:24.4580288Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:24.4614471Z Entering 'third_party/ittapi' 2024-06-26T07:04:24.4642015Z Entering 'third_party/kineto' 2024-06-26T07:04:24.4670135Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:24.4698847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:24.4729174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:24.4756981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:24.4786001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:24.4813941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:24.4842150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:24.4871876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:24.4900900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:24.4932547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:24.4960652Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:24.4989734Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:24.5017347Z Entering 'third_party/mimalloc' 2024-06-26T07:04:24.5048117Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:24.5075512Z Entering 'third_party/nlohmann' 2024-06-26T07:04:24.5103552Z Entering 'third_party/onnx' 2024-06-26T07:04:24.5144544Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:24.5172992Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:24.5202428Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:24.5234364Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:24.5263074Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:24.5290496Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:24.5319481Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:24.5348713Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:24.5375368Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:24.5403727Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:24.5430493Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:24.5458973Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:24.5490061Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:24.5533343Z Entering 'third_party/pocketfft' 2024-06-26T07:04:24.5561817Z Entering 'third_party/protobuf' 2024-06-26T07:04:24.5592297Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:24.5620759Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:24.5652007Z Entering 'third_party/psimd' 2024-06-26T07:04:24.5678538Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:24.5704508Z Entering 'third_party/pybind11' 2024-06-26T07:04:24.5732826Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:24.5759587Z Entering 'third_party/sleef' 2024-06-26T07:04:24.5788203Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:24.5816179Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:24.5842842Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:24.5868844Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:24.5896733Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:24.5923686Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:24.5964474Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:04:24.6175036Z Entering 'android/libs/fbjni' 2024-06-26T07:04:24.6202131Z Entering 'third_party/FP16' 2024-06-26T07:04:24.6228950Z Entering 'third_party/FXdiv' 2024-06-26T07:04:24.6255841Z Entering 'third_party/NNPACK' 2024-06-26T07:04:24.6283426Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:24.6311946Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:24.6348931Z Entering 'third_party/benchmark' 2024-06-26T07:04:24.6377597Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:24.6403395Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:24.6431888Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:24.6460059Z Entering 'third_party/cutlass' 2024-06-26T07:04:24.6495097Z Entering 'third_party/eigen' 2024-06-26T07:04:24.6522955Z Entering 'third_party/fbgemm' 2024-06-26T07:04:24.6550196Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:24.6577010Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:24.6604319Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:24.6635855Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:24.6662348Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:24.6691648Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:24.6720595Z Entering 'third_party/fmt' 2024-06-26T07:04:24.6747938Z Entering 'third_party/foxi' 2024-06-26T07:04:24.6777291Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:24.6805897Z Entering 'third_party/gloo' 2024-06-26T07:04:24.6834235Z Entering 'third_party/googletest' 2024-06-26T07:04:24.6862930Z Entering 'third_party/ideep' 2024-06-26T07:04:24.6891391Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:24.6926145Z Entering 'third_party/ittapi' 2024-06-26T07:04:24.6955358Z Entering 'third_party/kineto' 2024-06-26T07:04:24.6982748Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:24.7011608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:24.7040988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:24.7069577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:24.7097618Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:24.7125048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:24.7154708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:24.7181559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:24.7207627Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:24.7234129Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:24.7262433Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:24.7290785Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:24.7318223Z Entering 'third_party/mimalloc' 2024-06-26T07:04:24.7345369Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:24.7374904Z Entering 'third_party/nlohmann' 2024-06-26T07:04:24.7402241Z Entering 'third_party/onnx' 2024-06-26T07:04:24.7440231Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:24.7471843Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:24.7498546Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:24.7529964Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:24.7557610Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:24.7586525Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:24.7611818Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:24.7640905Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:24.7666751Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:24.7692805Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:24.7718586Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:24.7747046Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:24.7775922Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:24.7817384Z Entering 'third_party/pocketfft' 2024-06-26T07:04:24.7844655Z Entering 'third_party/protobuf' 2024-06-26T07:04:24.7876146Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:24.7903648Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:24.7934372Z Entering 'third_party/psimd' 2024-06-26T07:04:24.7960247Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:24.7987720Z Entering 'third_party/pybind11' 2024-06-26T07:04:24.8015163Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:24.8045528Z Entering 'third_party/sleef' 2024-06-26T07:04:24.8074334Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:24.8102627Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:24.8131942Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:24.8158286Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:24.8186841Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:24.8213257Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:24.8252014Z ##[endgroup] 2024-06-26T07:04:24.8279887Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:04:24.8296564Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:04:24.8464554Z Prepare all required actions 2024-06-26T07:04:24.8465020Z Getting action download info 2024-06-26T07:04:24.9782906Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:04:24.9783340Z env: 2024-06-26T07:04:24.9783623Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:24.9784122Z ##[endgroup] 2024-06-26T07:04:24.9853466Z ##[group]Run set -euo pipefail 2024-06-26T07:04:24.9853913Z set -euo pipefail 2024-06-26T07:04:24.9854310Z function get_ec2_metadata() { 2024-06-26T07:04:24.9854860Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:04:24.9855755Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:04:24.9856533Z  category=$1 2024-06-26T07:04:24.9857083Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:04:24.9857741Z  runner_name_str=i-02e64d1cff9e6c2ad 2024-06-26T07:04:24.9858246Z  if [[ -f /.inarc ]]; then 2024-06-26T07:04:24.9858749Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:04:24.9859338Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:04:24.9860048Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:04:24.9860661Z  else 2024-06-26T07:04:24.9861156Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:04:24.9861749Z  fi 2024-06-26T07:04:24.9862043Z } 2024-06-26T07:04:24.9862418Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:04:24.9863018Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:04:24.9863702Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:04:24.9864448Z echo "system info $(uname -a)" 2024-06-26T07:04:24.9872287Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:24.9872788Z env: 2024-06-26T07:04:24.9873082Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:24.9873441Z ##[endgroup] 2024-06-26T07:04:24.9941871Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:04:24.9985533Z instance-id: i-02e64d1cff9e6c2ad 2024-06-26T07:04:25.0028857Z instance-type: g5.4xlarge 2024-06-26T07:04:25.0035030Z system info Linux ip-10-0-19-247.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:04:25.0055237Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:25.0056559Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:25.0065211Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:25.0065825Z env: 2024-06-26T07:04:25.0066111Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:25.0066469Z ##[endgroup] 2024-06-26T07:04:25.0127177Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:04:25.0127800Z if systemctl is-active --quiet docker; then 2024-06-26T07:04:25.0128345Z  echo "Docker daemon is running..."; 2024-06-26T07:04:25.0128808Z else 2024-06-26T07:04:25.0129317Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:04:25.0129915Z fi 2024-06-26T07:04:25.0136787Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:25.0137293Z env: 2024-06-26T07:04:25.0137568Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:25.0137929Z ##[endgroup] 2024-06-26T07:04:25.0173815Z Docker daemon is running... 2024-06-26T07:04:25.0218325Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:04:25.0218890Z with: 2024-06-26T07:04:25.0219167Z shell: bash 2024-06-26T07:04:25.0219484Z timeout_minutes: 5 2024-06-26T07:04:25.0219818Z max_attempts: 3 2024-06-26T07:04:25.0220142Z retry_wait_seconds: 30 2024-06-26T07:04:25.0221661Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:04:25.0223106Z polling_interval_seconds: 1 2024-06-26T07:04:25.0223488Z warning_on_retry: true 2024-06-26T07:04:25.0224011Z continue_on_error: false 2024-06-26T07:04:25.0224360Z env: 2024-06-26T07:04:25.0224639Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:25.0225004Z AWS_RETRY_MODE: standard 2024-06-26T07:04:25.0225365Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:04:25.0225816Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:04:25.0226194Z ##[endgroup] 2024-06-26T07:04:25.9019651Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:25.9020494Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:25.9021338Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:25.9021989Z 2024-06-26T07:04:25.9022151Z Login Succeeded 2024-06-26T07:04:26.0716919Z Command completed after 1 attempt(s). 2024-06-26T07:04:26.0758614Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:26.0759384Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:26.0760072Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:26.0768280Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:26.0768784Z env: 2024-06-26T07:04:26.0769070Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.0769440Z ##[endgroup] 2024-06-26T07:04:26.0828075Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:26.0828856Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:26.0829475Z # shellcheck disable=SC2046 2024-06-26T07:04:26.0829943Z docker stop $(docker ps -q) || true 2024-06-26T07:04:26.0830443Z # Prune all of the docker images 2024-06-26T07:04:26.0830905Z docker system prune -af 2024-06-26T07:04:26.0837573Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:26.0838086Z env: 2024-06-26T07:04:26.0838369Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.0838731Z ##[endgroup] 2024-06-26T07:04:26.1083514Z "docker stop" requires at least 1 argument. 2024-06-26T07:04:26.1084128Z See 'docker stop --help'. 2024-06-26T07:04:26.1084372Z 2024-06-26T07:04:26.1085139Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:04:26.1085562Z 2024-06-26T07:04:26.1085714Z Stop one or more running containers 2024-06-26T07:04:26.1233355Z Total reclaimed space: 0B 2024-06-26T07:04:26.1262410Z ##[group]Run set +e 2024-06-26T07:04:26.1262782Z set +e 2024-06-26T07:04:26.1263212Z set -x 2024-06-26T07:04:26.1263510Z  2024-06-26T07:04:26.1264137Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:26.1264981Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:04:26.1266065Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:04:26.1266828Z # one is returned at random 2024-06-26T07:04:26.1267374Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:04:26.1267887Z  2024-06-26T07:04:26.1268214Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:26.1268837Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:04:26.1269656Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:04:26.1270213Z  2024-06-26T07:04:26.1270543Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:26.1271100Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:04:26.1271612Z  exit 1 2024-06-26T07:04:26.1271925Z  fi 2024-06-26T07:04:26.1272212Z fi 2024-06-26T07:04:26.1272628Z  2024-06-26T07:04:26.1272996Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:04:26.1273517Z  # Clean up any old records first 2024-06-26T07:04:26.1274045Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:04:26.1274508Z fi 2024-06-26T07:04:26.1274777Z  2024-06-26T07:04:26.1275212Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:04:26.1275762Z cat /etc/hosts 2024-06-26T07:04:26.1283453Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:26.1283953Z env: 2024-06-26T07:04:26.1284234Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.1284590Z ##[endgroup] 2024-06-26T07:04:26.1304152Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:26.1308106Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:04:26.1308537Z ++ tail -n1 2024-06-26T07:04:26.1644682Z + RESOLVED_IP=18.160.10.36 2024-06-26T07:04:26.1645279Z + '[' -z 18.160.10.36 ']' 2024-06-26T07:04:26.1645718Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:04:26.1651038Z 18.67.76.37 download.pytorch.org 2024-06-26T07:04:26.1652162Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:04:26.1750293Z + echo '18.160.10.36 download.pytorch.org' 2024-06-26T07:04:26.1750922Z + sudo tee -a /etc/hosts 2024-06-26T07:04:26.2095626Z 18.160.10.36 download.pytorch.org 2024-06-26T07:04:26.2108054Z + cat /etc/hosts 2024-06-26T07:04:26.2111947Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:04:26.2125297Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:04:26.2125809Z 18.160.10.36 download.pytorch.org 2024-06-26T07:04:26.2275931Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:04:26.2276601Z with: 2024-06-26T07:04:26.2277865Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2279154Z docker-build-dir: .ci/docker 2024-06-26T07:04:26.2279582Z working-directory: . 2024-06-26T07:04:26.2280160Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.2315620Z force-push: false 2024-06-26T07:04:26.2315949Z env: 2024-06-26T07:04:26.2316240Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.2316598Z ##[endgroup] 2024-06-26T07:04:26.2334415Z ##[group]Run set -ex 2024-06-26T07:04:26.2334775Z set -ex 2024-06-26T07:04:26.2335066Z  2024-06-26T07:04:26.2335629Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:04:26.2336610Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:04:26.2337538Z # job could then download the pre-built image as usual 2024-06-26T07:04:26.2338276Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:04:26.2338949Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2339584Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2340138Z  2024-06-26T07:04:26.2340654Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:04:26.2341277Z  exit 0 2024-06-26T07:04:26.2341571Z else 2024-06-26T07:04:26.2341935Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2342380Z fi 2024-06-26T07:04:26.2342648Z  2024-06-26T07:04:26.2343117Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:04:26.2344190Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:04:26.2345043Z  # use it as it is, but first let's extract the tag 2024-06-26T07:04:26.2345745Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:04:26.2346463Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2347159Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2347710Z else 2024-06-26T07:04:26.2348140Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:04:26.2348810Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2349706Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2350460Z fi 2024-06-26T07:04:26.2358148Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:26.2358645Z env: 2024-06-26T07:04:26.2358925Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.2359282Z REPO_NAME: pytorch 2024-06-26T07:04:26.2360405Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2361607Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:26.2362120Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.2362642Z ##[endgroup] 2024-06-26T07:04:26.2381784Z + [[ ! -d .ci/docker ]] 2024-06-26T07:04:26.2382347Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:04:26.2382856Z + echo skip=false 2024-06-26T07:04:26.2385031Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:04:26.2387957Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2389161Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:04:26.2394151Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2395231Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2396691Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2423018Z ##[group]Run set +e 2024-06-26T07:04:26.2423406Z set +e 2024-06-26T07:04:26.2423715Z set -x 2024-06-26T07:04:26.2424214Z  2024-06-26T07:04:26.2424506Z login() { 2024-06-26T07:04:26.2425194Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:26.2425924Z } 2024-06-26T07:04:26.2426202Z  2024-06-26T07:04:26.2426605Z retry () { 2024-06-26T07:04:26.2427009Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:26.2427461Z } 2024-06-26T07:04:26.2427739Z  2024-06-26T07:04:26.2428070Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:26.2428499Z  2024-06-26T07:04:26.2428981Z # Check if image already exists, if it does then skip building it 2024-06-26T07:04:26.2429706Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:04:26.2430227Z  exit 0 2024-06-26T07:04:26.2430540Z fi 2024-06-26T07:04:26.2430816Z  2024-06-26T07:04:26.2431329Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:04:26.2432171Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:04:26.2432925Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:04:26.2433597Z  # if we're on the base branch then use the parent commit 2024-06-26T07:04:26.2434207Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:04:26.2434647Z else 2024-06-26T07:04:26.2435136Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:04:26.2435840Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:04:26.2436357Z fi 2024-06-26T07:04:26.2436640Z  2024-06-26T07:04:26.2436976Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:04:26.2437480Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2437952Z  2024-06-26T07:04:26.2438616Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:04:26.2439391Z  exit 0 2024-06-26T07:04:26.2439689Z fi 2024-06-26T07:04:26.2439979Z  2024-06-26T07:04:26.2440439Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:04:26.2441431Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:04:26.2442232Z  exit 1 2024-06-26T07:04:26.2442554Z fi 2024-06-26T07:04:26.2442850Z  2024-06-26T07:04:26.2443352Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:04:26.2444318Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:04:26.2445204Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:04:26.2446267Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:04:26.2447392Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:04:26.2448146Z fi 2024-06-26T07:04:26.2448431Z  2024-06-26T07:04:26.2448814Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:26.2456473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:26.2456986Z env: 2024-06-26T07:04:26.2457275Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.2457669Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:26.2458161Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:26.2459473Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2460738Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.2461357Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.2461882Z ##[endgroup] 2024-06-26T07:04:26.2481512Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.2482172Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.2483069Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:26.2483844Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.6590491Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:26.6591322Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:26.6592195Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:26.6592803Z 2024-06-26T07:04:26.6592968Z Login Succeeded 2024-06-26T07:04:26.6603359Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.8727839Z { 2024-06-26T07:04:26.8728210Z "schemaVersion": 2, 2024-06-26T07:04:26.8728911Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:04:26.8729639Z "config": { 2024-06-26T07:04:26.8730111Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:04:26.8732465Z "size": 48681, 2024-06-26T07:04:26.8733063Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:04:26.8733884Z }, 2024-06-26T07:04:26.8734191Z "layers": [ 2024-06-26T07:04:26.8734481Z { 2024-06-26T07:04:26.8734936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8735503Z "size": 28580681, 2024-06-26T07:04:26.8736066Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:04:26.8736691Z }, 2024-06-26T07:04:26.8736947Z { 2024-06-26T07:04:26.8737445Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8738232Z "size": 7943451, 2024-06-26T07:04:26.8739116Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:04:26.8739979Z }, 2024-06-26T07:04:26.8740381Z { 2024-06-26T07:04:26.8740979Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8741725Z "size": 55728572, 2024-06-26T07:04:26.8742340Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:04:26.8742958Z }, 2024-06-26T07:04:26.8743216Z { 2024-06-26T07:04:26.8743653Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8744513Z "size": 186, 2024-06-26T07:04:26.8745081Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:04:26.8745710Z }, 2024-06-26T07:04:26.8745958Z { 2024-06-26T07:04:26.8746394Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8746957Z "size": 6886, 2024-06-26T07:04:26.8747506Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:04:26.8748128Z }, 2024-06-26T07:04:26.8748569Z { 2024-06-26T07:04:26.8749028Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8749586Z "size": 1329001756, 2024-06-26T07:04:26.8750174Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:04:26.8750809Z }, 2024-06-26T07:04:26.8751059Z { 2024-06-26T07:04:26.8751508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8752086Z "size": 62501, 2024-06-26T07:04:26.8752649Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:04:26.8753259Z }, 2024-06-26T07:04:26.8753515Z { 2024-06-26T07:04:26.8753960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8754517Z "size": 1684, 2024-06-26T07:04:26.8755051Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:04:26.8755677Z }, 2024-06-26T07:04:26.8755932Z { 2024-06-26T07:04:26.8756385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8757037Z "size": 1523, 2024-06-26T07:04:26.8757616Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:04:26.8758270Z }, 2024-06-26T07:04:26.8758521Z { 2024-06-26T07:04:26.8758967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8759576Z "size": 2528295403, 2024-06-26T07:04:26.8760168Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:04:26.8760778Z }, 2024-06-26T07:04:26.8761037Z { 2024-06-26T07:04:26.8761484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8762043Z "size": 86016, 2024-06-26T07:04:26.8762604Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:04:26.8763236Z }, 2024-06-26T07:04:26.8763498Z { 2024-06-26T07:04:26.8763941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8764500Z "size": 1823, 2024-06-26T07:04:26.8765047Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:04:26.8765666Z }, 2024-06-26T07:04:26.8766011Z { 2024-06-26T07:04:26.8766476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8767056Z "size": 246737542, 2024-06-26T07:04:26.8767620Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:04:26.8768222Z }, 2024-06-26T07:04:26.8768487Z { 2024-06-26T07:04:26.8768926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8769489Z "size": 545, 2024-06-26T07:04:26.8770035Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:04:26.8770666Z }, 2024-06-26T07:04:26.8770922Z { 2024-06-26T07:04:26.8771354Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8771926Z "size": 1283, 2024-06-26T07:04:26.8772479Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:04:26.8773105Z }, 2024-06-26T07:04:26.8773358Z { 2024-06-26T07:04:26.8773799Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8774371Z "size": 485, 2024-06-26T07:04:26.8774928Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:04:26.8775545Z }, 2024-06-26T07:04:26.8775805Z { 2024-06-26T07:04:26.8776258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8776828Z "size": 91710937, 2024-06-26T07:04:26.8777422Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:04:26.8778063Z }, 2024-06-26T07:04:26.8778328Z { 2024-06-26T07:04:26.8778766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8779347Z "size": 3233, 2024-06-26T07:04:26.8779980Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:04:26.8780615Z }, 2024-06-26T07:04:26.8780869Z { 2024-06-26T07:04:26.8781321Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8781895Z "size": 1905, 2024-06-26T07:04:26.8782478Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:04:26.8783100Z }, 2024-06-26T07:04:26.8783353Z { 2024-06-26T07:04:26.8783791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8784546Z "size": 702, 2024-06-26T07:04:26.8785101Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:26.8785734Z }, 2024-06-26T07:04:26.8785988Z { 2024-06-26T07:04:26.8786427Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8787001Z "size": 2784789002, 2024-06-26T07:04:26.8787587Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:04:26.8788288Z }, 2024-06-26T07:04:26.8788538Z { 2024-06-26T07:04:26.8788985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8789555Z "size": 381, 2024-06-26T07:04:26.8790093Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:04:26.8790717Z }, 2024-06-26T07:04:26.8790979Z { 2024-06-26T07:04:26.8791423Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8791978Z "size": 11879, 2024-06-26T07:04:26.8792522Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:04:26.8793138Z }, 2024-06-26T07:04:26.8793392Z { 2024-06-26T07:04:26.8793824Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8794392Z "size": 803, 2024-06-26T07:04:26.8794959Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:04:26.8795581Z }, 2024-06-26T07:04:26.8795843Z { 2024-06-26T07:04:26.8796298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8796879Z "size": 106, 2024-06-26T07:04:26.8797430Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:04:26.8798052Z }, 2024-06-26T07:04:26.8798316Z { 2024-06-26T07:04:26.8798777Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8799348Z "size": 503, 2024-06-26T07:04:26.8799946Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:04:26.8800585Z }, 2024-06-26T07:04:26.8800844Z { 2024-06-26T07:04:26.8801289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8801863Z "size": 121477177, 2024-06-26T07:04:26.8802448Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:04:26.8803075Z }, 2024-06-26T07:04:26.8803326Z { 2024-06-26T07:04:26.8803783Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8804365Z "size": 109, 2024-06-26T07:04:26.8804908Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:04:26.8805540Z }, 2024-06-26T07:04:26.8805812Z { 2024-06-26T07:04:26.8806358Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8806932Z "size": 491, 2024-06-26T07:04:26.8807481Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:04:26.8808104Z }, 2024-06-26T07:04:26.8808364Z { 2024-06-26T07:04:26.8808807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8809384Z "size": 291, 2024-06-26T07:04:26.8809963Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:04:26.8810587Z }, 2024-06-26T07:04:26.8810932Z { 2024-06-26T07:04:26.8811388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8811962Z "size": 101, 2024-06-26T07:04:26.8812507Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:04:26.8813139Z }, 2024-06-26T07:04:26.8813406Z { 2024-06-26T07:04:26.8813861Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8814424Z "size": 1473, 2024-06-26T07:04:26.8814983Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:04:26.8815591Z }, 2024-06-26T07:04:26.8815844Z { 2024-06-26T07:04:26.8816301Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8816887Z "size": 446710781, 2024-06-26T07:04:26.8817464Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:04:26.8818085Z }, 2024-06-26T07:04:26.8818350Z { 2024-06-26T07:04:26.8818810Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8819455Z "size": 161, 2024-06-26T07:04:26.8819990Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:04:26.8820607Z }, 2024-06-26T07:04:26.8820875Z { 2024-06-26T07:04:26.8821337Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8821903Z "size": 564, 2024-06-26T07:04:26.8822458Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:04:26.8823085Z }, 2024-06-26T07:04:26.8823336Z { 2024-06-26T07:04:26.8824106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8824865Z "size": 35874600, 2024-06-26T07:04:26.8825576Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:04:26.8826381Z }, 2024-06-26T07:04:26.8826757Z { 2024-06-26T07:04:26.8827455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8828311Z "size": 104, 2024-06-26T07:04:26.8829120Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:04:26.8829996Z }, 2024-06-26T07:04:26.8830327Z { 2024-06-26T07:04:26.8830916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8831664Z "size": 425, 2024-06-26T07:04:26.8832361Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:04:26.8833097Z }, 2024-06-26T07:04:26.8833357Z { 2024-06-26T07:04:26.8833808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8834367Z "size": 20262084, 2024-06-26T07:04:26.8834949Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:04:26.8835582Z }, 2024-06-26T07:04:26.8835841Z { 2024-06-26T07:04:26.8836284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8836861Z "size": 439, 2024-06-26T07:04:26.8837397Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:04:26.8838014Z }, 2024-06-26T07:04:26.8838265Z { 2024-06-26T07:04:26.8838712Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8839278Z "size": 702, 2024-06-26T07:04:26.8839879Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:26.8840508Z }, 2024-06-26T07:04:26.8840768Z { 2024-06-26T07:04:26.8841217Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8841772Z "size": 143, 2024-06-26T07:04:26.8842314Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:04:26.8842938Z }, 2024-06-26T07:04:26.8843203Z { 2024-06-26T07:04:26.8843638Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8844211Z "size": 135, 2024-06-26T07:04:26.8844896Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:04:26.8845529Z }, 2024-06-26T07:04:26.8845795Z { 2024-06-26T07:04:26.8846320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8846895Z "size": 4911495247, 2024-06-26T07:04:26.8847457Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:04:26.8848075Z }, 2024-06-26T07:04:26.8848338Z { 2024-06-26T07:04:26.8848788Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8849355Z "size": 190, 2024-06-26T07:04:26.8849973Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:04:26.8850614Z }, 2024-06-26T07:04:26.8850876Z { 2024-06-26T07:04:26.8851318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8851887Z "size": 565, 2024-06-26T07:04:26.8852439Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:04:26.8853121Z }, 2024-06-26T07:04:26.8853388Z { 2024-06-26T07:04:26.8853833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8854408Z "size": 43163095, 2024-06-26T07:04:26.8854970Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:04:26.8855606Z }, 2024-06-26T07:04:26.8855870Z { 2024-06-26T07:04:26.8856319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8856875Z "size": 106, 2024-06-26T07:04:26.8857431Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:04:26.8858062Z }, 2024-06-26T07:04:26.8858314Z { 2024-06-26T07:04:26.8858764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8859339Z "size": 1213, 2024-06-26T07:04:26.8859914Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:04:26.8860528Z }, 2024-06-26T07:04:26.8860798Z { 2024-06-26T07:04:26.8861250Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8861824Z "size": 702, 2024-06-26T07:04:26.8862368Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:26.8863000Z }, 2024-06-26T07:04:26.8863265Z { 2024-06-26T07:04:26.8863711Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8864485Z "size": 138, 2024-06-26T07:04:26.8865059Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:04:26.8865710Z }, 2024-06-26T07:04:26.8865971Z { 2024-06-26T07:04:26.8866431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8867011Z "size": 120, 2024-06-26T07:04:26.8867577Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:04:26.8868217Z }, 2024-06-26T07:04:26.8868495Z { 2024-06-26T07:04:26.8868950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8869539Z "size": 1904157495, 2024-06-26T07:04:26.8870120Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:04:26.8870761Z }, 2024-06-26T07:04:26.8871022Z { 2024-06-26T07:04:26.8871475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8872052Z "size": 172, 2024-06-26T07:04:26.8872613Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:04:26.8873260Z }, 2024-06-26T07:04:26.8873519Z { 2024-06-26T07:04:26.8873979Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8874559Z "size": 908, 2024-06-26T07:04:26.8875147Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:04:26.8875780Z }, 2024-06-26T07:04:26.8876148Z { 2024-06-26T07:04:26.8876605Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8877224Z + exit 0 2024-06-26T07:04:26.8877525Z "size": 702, 2024-06-26T07:04:26.8878077Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:26.8878721Z }, 2024-06-26T07:04:26.8878988Z { 2024-06-26T07:04:26.8879433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8880007Z "size": 134, 2024-06-26T07:04:26.8880555Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:04:26.8881189Z }, 2024-06-26T07:04:26.8881448Z { 2024-06-26T07:04:26.8881913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8882490Z "size": 32, 2024-06-26T07:04:26.8883058Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:26.8883684Z }, 2024-06-26T07:04:26.8883961Z { 2024-06-26T07:04:26.8884426Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8885067Z "size": 156, 2024-06-26T07:04:26.8885625Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:04:26.8886339Z }, 2024-06-26T07:04:26.8886607Z { 2024-06-26T07:04:26.8887046Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8887617Z "size": 1841, 2024-06-26T07:04:26.8888170Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:04:26.8888793Z }, 2024-06-26T07:04:26.8889041Z { 2024-06-26T07:04:26.8889482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8890049Z "size": 7529784, 2024-06-26T07:04:26.8890605Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:04:26.8891217Z }, 2024-06-26T07:04:26.8891471Z { 2024-06-26T07:04:26.8891913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8892469Z "size": 165, 2024-06-26T07:04:26.8893014Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:04:26.8893632Z }, 2024-06-26T07:04:26.8893885Z { 2024-06-26T07:04:26.8894318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8894881Z "size": 7944, 2024-06-26T07:04:26.8895509Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:04:26.8896131Z }, 2024-06-26T07:04:26.8896378Z { 2024-06-26T07:04:26.8896823Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8897384Z "size": 8070, 2024-06-26T07:04:26.8897929Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:04:26.8898533Z }, 2024-06-26T07:04:26.8898786Z { 2024-06-26T07:04:26.8899233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8899785Z "size": 303, 2024-06-26T07:04:26.8900341Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:04:26.8900964Z }, 2024-06-26T07:04:26.8901228Z { 2024-06-26T07:04:26.8901659Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8902227Z "size": 7629071, 2024-06-26T07:04:26.8902762Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:04:26.8903360Z }, 2024-06-26T07:04:26.8903615Z { 2024-06-26T07:04:26.8904239Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8904808Z "size": 108, 2024-06-26T07:04:26.8905348Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:04:26.8905974Z }, 2024-06-26T07:04:26.8906227Z { 2024-06-26T07:04:26.8906668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8907328Z "size": 54145775, 2024-06-26T07:04:26.8907904Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:04:26.8908533Z }, 2024-06-26T07:04:26.8908796Z { 2024-06-26T07:04:26.8909231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8909793Z "size": 474, 2024-06-26T07:04:26.8910341Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:04:26.8910960Z }, 2024-06-26T07:04:26.8911207Z { 2024-06-26T07:04:26.8911650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8912212Z "size": 1374858747, 2024-06-26T07:04:26.8912766Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:04:26.8913366Z }, 2024-06-26T07:04:26.8913627Z { 2024-06-26T07:04:26.8914066Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8914614Z "size": 106, 2024-06-26T07:04:26.8915174Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:04:26.8915936Z }, 2024-06-26T07:04:26.8916193Z { 2024-06-26T07:04:26.8916624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8917184Z "size": 558, 2024-06-26T07:04:26.8917734Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:04:26.8918366Z }, 2024-06-26T07:04:26.8918613Z { 2024-06-26T07:04:26.8919057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8919661Z "size": 46248513, 2024-06-26T07:04:26.8920220Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:04:26.8920830Z }, 2024-06-26T07:04:26.8921082Z { 2024-06-26T07:04:26.8921523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8922075Z "size": 111, 2024-06-26T07:04:26.8922627Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:04:26.8923252Z }, 2024-06-26T07:04:26.8923511Z { 2024-06-26T07:04:26.8923946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8924508Z "size": 32, 2024-06-26T07:04:26.8925055Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:26.8925672Z }, 2024-06-26T07:04:26.8925994Z { 2024-06-26T07:04:26.8926444Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8927005Z "size": 32, 2024-06-26T07:04:26.8927548Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:26.8928179Z }, 2024-06-26T07:04:26.8928440Z { 2024-06-26T07:04:26.8928885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8929435Z "size": 32, 2024-06-26T07:04:26.8929986Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:26.8930618Z }, 2024-06-26T07:04:26.8930867Z { 2024-06-26T07:04:26.8931313Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:26.8931877Z "size": 32, 2024-06-26T07:04:26.8932416Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:26.8933032Z } 2024-06-26T07:04:26.8933294Z ] 2024-06-26T07:04:26.8933548Z } 2024-06-26T07:04:26.9053740Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:26.9054240Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:26.9054804Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:04:26.9062993Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:26.9063505Z env: 2024-06-26T07:04:26.9063792Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.9065102Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.9066274Z ##[endgroup] 2024-06-26T07:04:26.9088860Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.9139063Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:04:26.9139645Z with: 2024-06-26T07:04:26.9140811Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.9142137Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.9142666Z env: 2024-06-26T07:04:26.9142962Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.9143331Z ##[endgroup] 2024-06-26T07:04:26.9159139Z ##[group]Run set -x 2024-06-26T07:04:26.9159498Z set -x 2024-06-26T07:04:26.9159815Z set +e 2024-06-26T07:04:26.9160122Z  2024-06-26T07:04:26.9160403Z login() { 2024-06-26T07:04:26.9161084Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:26.9161933Z } 2024-06-26T07:04:26.9162224Z  2024-06-26T07:04:26.9162534Z retry () { 2024-06-26T07:04:26.9162940Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:26.9163397Z } 2024-06-26T07:04:26.9163687Z  2024-06-26T07:04:26.9164015Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:26.9164436Z  2024-06-26T07:04:26.9164723Z set -e 2024-06-26T07:04:26.9165210Z # ignore output since only exit code is used for conditional 2024-06-26T07:04:26.9166018Z # only pull docker image if it's not available locally 2024-06-26T07:04:26.9166800Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:04:26.9167512Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:04:26.9167971Z fi 2024-06-26T07:04:26.9175754Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:26.9176284Z env: 2024-06-26T07:04:26.9176594Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:26.9177769Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:26.9179092Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.9179643Z ##[endgroup] 2024-06-26T07:04:26.9198662Z + set +e 2024-06-26T07:04:26.9199166Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.9199803Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:26.9200703Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:26.9201668Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:27.3212300Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:27.3213187Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:27.3214038Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:27.3214585Z 2024-06-26T07:04:27.3214704Z Login Succeeded 2024-06-26T07:04:27.3222387Z + set -e 2024-06-26T07:04:27.3223764Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:27.3345278Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:27.3347721Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:27.5445489Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:04:27.5448052Z 7a2c55901189: Pulling fs layer 2024-06-26T07:04:27.5449077Z 224fe954d725: Pulling fs layer 2024-06-26T07:04:27.5450176Z 75722010b82e: Pulling fs layer 2024-06-26T07:04:27.5450670Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:04:27.5451057Z b57676e46aee: Pulling fs layer 2024-06-26T07:04:27.5451445Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:04:27.5451825Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:04:27.5452302Z 0c1227890755: Pulling fs layer 2024-06-26T07:04:27.5452814Z d8d1234baab3: Pulling fs layer 2024-06-26T07:04:27.5453296Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:04:27.5453713Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:04:27.5454100Z 309b872da805: Pulling fs layer 2024-06-26T07:04:27.5454475Z e82932cb002c: Pulling fs layer 2024-06-26T07:04:27.5454913Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:04:27.5455429Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:04:27.5455937Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:04:27.5456405Z f4863801e30c: Pulling fs layer 2024-06-26T07:04:27.5456936Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:04:27.5457372Z a8c1e85b5e14: Waiting 2024-06-26T07:04:27.5457750Z cf2f34f225db: Pulling fs layer 2024-06-26T07:04:27.5458224Z a41a8d1c11c8: Waiting 2024-06-26T07:04:27.5458637Z df546d7b3e34: Pulling fs layer 2024-06-26T07:04:27.5459096Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:04:27.5459464Z 0c1227890755: Waiting 2024-06-26T07:04:27.5459891Z 96d0e2032184: Pulling fs layer 2024-06-26T07:04:27.5460386Z 06e12ce59790: Pulling fs layer 2024-06-26T07:04:27.5460840Z d8d1234baab3: Waiting 2024-06-26T07:04:27.5461167Z 7ed32bc8e469: Waiting 2024-06-26T07:04:27.5461501Z ed712f63cde7: Pulling fs layer 2024-06-26T07:04:27.5461868Z ec1e7978c1fe: Waiting 2024-06-26T07:04:27.5462209Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:04:27.5462564Z 309b872da805: Waiting 2024-06-26T07:04:27.5462951Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:04:27.5463340Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:04:27.5463699Z e82932cb002c: Waiting 2024-06-26T07:04:27.5464297Z ca9dd73e3e93: Waiting 2024-06-26T07:04:27.5464621Z d37dd9ed8948: Waiting 2024-06-26T07:04:27.5464984Z 16d76a79d5d6: Waiting 2024-06-26T07:04:27.5465334Z de45a14c9b22: Pulling fs layer 2024-06-26T07:04:27.5465738Z 4bdf41c9ecbe: Waiting 2024-06-26T07:04:27.5466055Z f4863801e30c: Waiting 2024-06-26T07:04:27.5466408Z d527cbbb87e3: Waiting 2024-06-26T07:04:27.5466753Z e99241e77209: Pulling fs layer 2024-06-26T07:04:27.5467140Z b57676e46aee: Waiting 2024-06-26T07:04:27.5467461Z cf2f34f225db: Waiting 2024-06-26T07:04:27.5467775Z 3d914950f8ad: Waiting 2024-06-26T07:04:27.5468109Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:04:27.5468466Z df546d7b3e34: Waiting 2024-06-26T07:04:27.5468800Z a823ffa4c541: Pulling fs layer 2024-06-26T07:04:27.5469159Z 96d0e2032184: Waiting 2024-06-26T07:04:27.5469468Z a97a71ef8a4e: Waiting 2024-06-26T07:04:27.5469814Z 06e12ce59790: Waiting 2024-06-26T07:04:27.5470164Z 44ba272eae20: Pulling fs layer 2024-06-26T07:04:27.5470527Z 25cd0c482ad6: Waiting 2024-06-26T07:04:27.5470845Z ed712f63cde7: Waiting 2024-06-26T07:04:27.5471179Z da02d278e524: Pulling fs layer 2024-06-26T07:04:27.5471573Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:04:27.5471940Z a823ffa4c541: Waiting 2024-06-26T07:04:27.5472266Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:04:27.5472635Z da02d278e524: Waiting 2024-06-26T07:04:27.5472969Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:04:27.5473327Z 9bf94bc45658: Waiting 2024-06-26T07:04:27.5473657Z b2950a036534: Pulling fs layer 2024-06-26T07:04:27.5474038Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:04:27.5474396Z b2950a036534: Waiting 2024-06-26T07:04:27.5474745Z de45a14c9b22: Waiting 2024-06-26T07:04:27.5475087Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:04:27.5475463Z 7a868239227f: Pulling fs layer 2024-06-26T07:04:27.5475831Z 25e19d81d9fc: Waiting 2024-06-26T07:04:27.5476141Z 60156b40fcbe: Waiting 2024-06-26T07:04:27.5476473Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:04:27.5476839Z e99241e77209: Waiting 2024-06-26T07:04:27.5477168Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:04:27.5477552Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:04:27.5477911Z 44ba272eae20: Waiting 2024-06-26T07:04:27.5478370Z 83911d7c7ecf: Waiting 2024-06-26T07:04:27.5478707Z c1068d03a811: Pulling fs layer 2024-06-26T07:04:27.5479070Z 84b2c7a89982: Waiting 2024-06-26T07:04:27.5479385Z b3c5d66c679f: Waiting 2024-06-26T07:04:27.5479697Z 9cd4c745f8e4: Waiting 2024-06-26T07:04:27.5480028Z 671b4a301d81: Pulling fs layer 2024-06-26T07:04:27.5480413Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:04:27.5480794Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:04:27.5481152Z c1068d03a811: Waiting 2024-06-26T07:04:27.5481483Z fb2e97768560: Pulling fs layer 2024-06-26T07:04:27.5481845Z fb2e97768560: Waiting 2024-06-26T07:04:27.5482179Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:04:27.5482552Z e08e88c13668: Pulling fs layer 2024-06-26T07:04:27.5482925Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:04:27.5483305Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:04:27.5483681Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:04:27.5484127Z b599c5a8c7cb: Waiting 2024-06-26T07:04:27.5484450Z cf7c2cb896cb: Waiting 2024-06-26T07:04:27.5484793Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:04:27.5485167Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:04:27.5485554Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:04:27.5486038Z e797fe86caef: Pulling fs layer 2024-06-26T07:04:27.5486425Z 967190076fc0: Pulling fs layer 2024-06-26T07:04:27.5486792Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:04:27.5487170Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:04:27.5487547Z cc91933e985a: Pulling fs layer 2024-06-26T07:04:27.5487917Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:04:27.5488277Z 4f4fb700ef54: Waiting 2024-06-26T07:04:27.5488604Z 526083994857: Pulling fs layer 2024-06-26T07:04:27.5488975Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:04:27.5489350Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:04:27.5489712Z e797fe86caef: Waiting 2024-06-26T07:04:27.5490036Z 4fb0a83f0685: Waiting 2024-06-26T07:04:27.5490350Z 967190076fc0: Waiting 2024-06-26T07:04:27.5490682Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:04:27.5491051Z 68af2ec4e30a: Waiting 2024-06-26T07:04:27.5491384Z 95237e7311a1: Pulling fs layer 2024-06-26T07:04:27.5491737Z cc91933e985a: Waiting 2024-06-26T07:04:27.5492051Z 9f3187916e7d: Waiting 2024-06-26T07:04:27.5492383Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:04:27.5492742Z 526083994857: Waiting 2024-06-26T07:04:27.5493062Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:04:27.5493440Z 2060028f7488: Pulling fs layer 2024-06-26T07:04:27.5493797Z 05a6f6dcc1c0: Waiting 2024-06-26T07:04:27.5494108Z 1b0e27c35f07: Waiting 2024-06-26T07:04:27.5494444Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:04:27.5494809Z 95237e7311a1: Waiting 2024-06-26T07:04:27.5495123Z de0f8a8c45c6: Waiting 2024-06-26T07:04:27.5495431Z 2060028f7488: Waiting 2024-06-26T07:04:27.5495740Z 789eff6c51aa: Waiting 2024-06-26T07:04:27.5496059Z e412c7f3b2de: Waiting 2024-06-26T07:04:27.6751026Z 224fe954d725: Verifying Checksum 2024-06-26T07:04:27.6751697Z 224fe954d725: Download complete 2024-06-26T07:04:27.7442245Z d527cbbb87e3: Download complete 2024-06-26T07:04:27.8207839Z b57676e46aee: Verifying Checksum 2024-06-26T07:04:27.8208422Z b57676e46aee: Download complete 2024-06-26T07:04:27.8869563Z 7a2c55901189: Verifying Checksum 2024-06-26T07:04:27.8870015Z 7a2c55901189: Download complete 2024-06-26T07:04:27.9567034Z a41a8d1c11c8: Download complete 2024-06-26T07:04:28.0198247Z 0c1227890755: Verifying Checksum 2024-06-26T07:04:28.0198682Z 0c1227890755: Download complete 2024-06-26T07:04:28.0955782Z d8d1234baab3: Verifying Checksum 2024-06-26T07:04:28.0956225Z d8d1234baab3: Download complete 2024-06-26T07:04:28.1927670Z 75722010b82e: Verifying Checksum 2024-06-26T07:04:28.1928118Z 75722010b82e: Download complete 2024-06-26T07:04:28.2616146Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:04:28.2616620Z ec1e7978c1fe: Download complete 2024-06-26T07:04:28.3632768Z 309b872da805: Verifying Checksum 2024-06-26T07:04:28.3633253Z 309b872da805: Download complete 2024-06-26T07:04:28.9185595Z 7a2c55901189: Pull complete 2024-06-26T07:04:29.2267206Z 224fe954d725: Pull complete 2024-06-26T07:04:30.1739919Z 75722010b82e: Pull complete 2024-06-26T07:04:30.2428909Z d527cbbb87e3: Pull complete 2024-06-26T07:04:30.2967999Z b57676e46aee: Pull complete 2024-06-26T07:04:30.8902987Z e82932cb002c: Verifying Checksum 2024-06-26T07:04:30.8903455Z e82932cb002c: Download complete 2024-06-26T07:04:30.9547628Z ca9dd73e3e93: Download complete 2024-06-26T07:04:31.0318697Z 16d76a79d5d6: Download complete 2024-06-26T07:04:31.1079794Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:04:31.1080319Z 4bdf41c9ecbe: Download complete 2024-06-26T07:04:32.0695159Z f4863801e30c: Verifying Checksum 2024-06-26T07:04:32.0695693Z f4863801e30c: Download complete 2024-06-26T07:04:32.1412648Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:04:32.1413251Z d37dd9ed8948: Download complete 2024-06-26T07:04:32.2383296Z cf2f34f225db: Verifying Checksum 2024-06-26T07:04:32.3109608Z df546d7b3e34: Verifying Checksum 2024-06-26T07:04:41.1596892Z a8c1e85b5e14: Download complete 2024-06-26T07:04:41.2338657Z 96d0e2032184: Verifying Checksum 2024-06-26T07:04:41.2339191Z 96d0e2032184: Download complete 2024-06-26T07:04:41.3131125Z 06e12ce59790: Verifying Checksum 2024-06-26T07:04:41.3131687Z 06e12ce59790: Download complete 2024-06-26T07:04:41.3868407Z ed712f63cde7: Verifying Checksum 2024-06-26T07:04:41.3869026Z ed712f63cde7: Download complete 2024-06-26T07:04:41.4690964Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:04:41.4691442Z a97a71ef8a4e: Download complete 2024-06-26T07:04:41.5476246Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:04:41.5476877Z 25cd0c482ad6: Download complete 2024-06-26T07:04:42.8564715Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:04:42.8565203Z 9bf94bc45658: Download complete 2024-06-26T07:04:42.9282764Z de45a14c9b22: Verifying Checksum 2024-06-26T07:04:42.9283368Z de45a14c9b22: Download complete 2024-06-26T07:04:43.0045371Z e99241e77209: Verifying Checksum 2024-06-26T07:04:43.0045852Z e99241e77209: Download complete 2024-06-26T07:04:43.0877007Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:04:43.0877616Z 35e8f4e2a42f: Download complete 2024-06-26T07:04:43.1734268Z a823ffa4c541: Verifying Checksum 2024-06-26T07:04:43.1734832Z a823ffa4c541: Download complete 2024-06-26T07:04:43.2550301Z 44ba272eae20: Download complete 2024-06-26T07:04:47.7870754Z da02d278e524: Verifying Checksum 2024-06-26T07:04:47.7871423Z da02d278e524: Download complete 2024-06-26T07:04:47.8711885Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:04:47.8712439Z 73a0ce681c29: Download complete 2024-06-26T07:04:47.9387882Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:04:47.9388987Z 49737bd5ae70: Download complete 2024-06-26T07:04:48.3514663Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:04:48.3515209Z 9cd4c745f8e4: Download complete 2024-06-26T07:04:48.4196475Z b2950a036534: Download complete 2024-06-26T07:04:48.4813142Z 25e19d81d9fc: Download complete 2024-06-26T07:04:48.7516174Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:04:48.7516817Z 60156b40fcbe: Download complete 2024-06-26T07:04:48.8578997Z 7a868239227f: Download complete 2024-06-26T07:04:48.9378182Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:04:48.9378854Z 83911d7c7ecf: Download complete 2024-06-26T07:04:49.0102503Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:04:49.0103084Z b3c5d66c679f: Download complete 2024-06-26T07:04:53.4345525Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:04:53.4346113Z 7ed32bc8e469: Download complete 2024-06-26T07:04:53.5080761Z c1068d03a811: Download complete 2024-06-26T07:04:53.5829642Z 671b4a301d81: Verifying Checksum 2024-06-26T07:04:53.5830232Z 671b4a301d81: Download complete 2024-06-26T07:04:53.8757481Z a8c1e85b5e14: Pull complete 2024-06-26T07:04:54.0573489Z a41a8d1c11c8: Pull complete 2024-06-26T07:04:54.0820404Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:04:54.0821004Z 67760cf5ec46: Download complete 2024-06-26T07:04:54.1551874Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:04:54.1552451Z ddc9740bcea2: Download complete 2024-06-26T07:04:54.2218211Z fb2e97768560: Download complete 2024-06-26T07:04:54.2744499Z 0c1227890755: Pull complete 2024-06-26T07:04:54.2958928Z b599c5a8c7cb: Download complete 2024-06-26T07:04:54.3668383Z e08e88c13668: Verifying Checksum 2024-06-26T07:04:54.3668984Z e08e88c13668: Download complete 2024-06-26T07:04:54.5143397Z d8d1234baab3: Pull complete 2024-06-26T07:05:00.2052241Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:05:00.2052708Z 3d914950f8ad: Download complete 2024-06-26T07:05:00.2810432Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:05:00.2811017Z cf7c2cb896cb: Download complete 2024-06-26T07:05:00.3539603Z 3a778ddfac2a: Download complete 2024-06-26T07:05:00.4408826Z 236c8f0245f4: Download complete 2024-06-26T07:05:00.4492754Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:05:00.4493294Z 4f4fb700ef54: Download complete 2024-06-26T07:05:00.5258150Z dda1ce302b9a: Download complete 2024-06-26T07:05:00.6467210Z e797fe86caef: Download complete 2024-06-26T07:05:00.7787388Z 967190076fc0: Verifying Checksum 2024-06-26T07:05:00.7788254Z 967190076fc0: Download complete 2024-06-26T07:05:00.9153231Z 68af2ec4e30a: Download complete 2024-06-26T07:05:00.9876068Z cc91933e985a: Download complete 2024-06-26T07:05:01.0604741Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:05:01.0605179Z 9f3187916e7d: Download complete 2024-06-26T07:05:01.1886769Z 526083994857: Verifying Checksum 2024-06-26T07:05:01.1887193Z 526083994857: Download complete 2024-06-26T07:05:01.2617034Z 05a6f6dcc1c0: Download complete 2024-06-26T07:05:01.8570804Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:05:01.8571412Z 6bc9dc8315b0: Download complete 2024-06-26T07:05:01.9616361Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:05:01.9616832Z 1b0e27c35f07: Download complete 2024-06-26T07:05:24.6304559Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:05:24.6305109Z 12ba7431ff38: Download complete 2024-06-26T07:05:24.7049007Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:05:24.7049584Z 789eff6c51aa: Download complete 2024-06-26T07:05:24.7922341Z de0f8a8c45c6: Download complete 2024-06-26T07:05:26.2755186Z 2060028f7488: Verifying Checksum 2024-06-26T07:05:26.2755851Z 2060028f7488: Download complete 2024-06-26T07:05:26.3525412Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:05:26.3526097Z e412c7f3b2de: Download complete 2024-06-26T07:05:28.7728402Z 95237e7311a1: Verifying Checksum 2024-06-26T07:05:28.7728870Z 95237e7311a1: Download complete 2024-06-26T07:05:52.8631893Z 84b2c7a89982: Download complete 2024-06-26T07:06:10.1901450Z 7ed32bc8e469: Pull complete 2024-06-26T07:06:10.4105095Z ec1e7978c1fe: Pull complete 2024-06-26T07:06:10.6516728Z 309b872da805: Pull complete 2024-06-26T07:06:18.0313215Z e82932cb002c: Pull complete 2024-06-26T07:06:18.2553359Z ca9dd73e3e93: Pull complete 2024-06-26T07:06:18.4701320Z 16d76a79d5d6: Pull complete 2024-06-26T07:06:18.7013572Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:06:21.1059402Z f4863801e30c: Pull complete 2024-06-26T07:06:21.3208183Z d37dd9ed8948: Pull complete 2024-06-26T07:06:21.5435031Z cf2f34f225db: Pull complete 2024-06-26T07:06:21.7706325Z df546d7b3e34: Pull complete 2024-06-26T07:07:10.8616422Z 3d914950f8ad: Pull complete 2024-06-26T07:07:11.0637751Z 96d0e2032184: Pull complete 2024-06-26T07:07:11.2699052Z 06e12ce59790: Pull complete 2024-06-26T07:07:11.4666604Z ed712f63cde7: Pull complete 2024-06-26T07:07:11.6178233Z a97a71ef8a4e: Pull complete 2024-06-26T07:07:11.7177726Z 25cd0c482ad6: Pull complete 2024-06-26T07:07:15.2164300Z 9bf94bc45658: Pull complete 2024-06-26T07:07:15.3406089Z de45a14c9b22: Pull complete 2024-06-26T07:07:15.5529900Z e99241e77209: Pull complete 2024-06-26T07:07:15.7755158Z 35e8f4e2a42f: Pull complete 2024-06-26T07:07:16.0059316Z a823ffa4c541: Pull complete 2024-06-26T07:07:16.2220142Z 44ba272eae20: Pull complete 2024-06-26T07:07:24.5074625Z da02d278e524: Pull complete 2024-06-26T07:07:24.7064278Z 73a0ce681c29: Pull complete 2024-06-26T07:07:24.9179806Z 49737bd5ae70: Pull complete 2024-06-26T07:07:25.9003531Z 9cd4c745f8e4: Pull complete 2024-06-26T07:07:26.0749964Z b2950a036534: Pull complete 2024-06-26T07:07:26.2263464Z 25e19d81d9fc: Pull complete 2024-06-26T07:07:26.6915482Z 60156b40fcbe: Pull complete 2024-06-26T07:07:26.8878148Z 7a868239227f: Pull complete 2024-06-26T07:07:27.2105936Z 83911d7c7ecf: Pull complete 2024-06-26T07:07:27.4434738Z b3c5d66c679f: Pull complete 2024-06-26T07:08:06.1145751Z 84b2c7a89982: Pull complete 2024-06-26T07:08:06.3387415Z c1068d03a811: Pull complete 2024-06-26T07:08:06.4923065Z 671b4a301d81: Pull complete 2024-06-26T07:08:08.4443157Z 67760cf5ec46: Pull complete 2024-06-26T07:08:08.5087213Z ddc9740bcea2: Pull complete 2024-06-26T07:08:08.5588661Z fb2e97768560: Pull complete 2024-06-26T07:08:08.6798235Z b599c5a8c7cb: Pull complete 2024-06-26T07:08:08.7266016Z e08e88c13668: Pull complete 2024-06-26T07:08:42.8750500Z 12ba7431ff38: Pull complete 2024-06-26T07:08:43.0151573Z cf7c2cb896cb: Pull complete 2024-06-26T07:08:43.2409843Z 3a778ddfac2a: Pull complete 2024-06-26T07:08:43.7164197Z 236c8f0245f4: Pull complete 2024-06-26T07:08:43.9384093Z 4f4fb700ef54: Pull complete 2024-06-26T07:08:44.1484905Z dda1ce302b9a: Pull complete 2024-06-26T07:08:44.3674135Z e797fe86caef: Pull complete 2024-06-26T07:08:44.7391394Z 967190076fc0: Pull complete 2024-06-26T07:08:44.9640496Z 4fb0a83f0685: Pull complete 2024-06-26T07:08:45.1822684Z 68af2ec4e30a: Pull complete 2024-06-26T07:08:45.3968525Z cc91933e985a: Pull complete 2024-06-26T07:08:45.6203559Z 9f3187916e7d: Pull complete 2024-06-26T07:08:46.6736445Z 526083994857: Pull complete 2024-06-26T07:08:46.8798958Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:08:48.9495418Z 6bc9dc8315b0: Pull complete 2024-06-26T07:08:49.1631566Z 1b0e27c35f07: Pull complete 2024-06-26T07:09:02.7479750Z 95237e7311a1: Pull complete 2024-06-26T07:09:02.8006865Z 789eff6c51aa: Pull complete 2024-06-26T07:09:02.8422450Z de0f8a8c45c6: Pull complete 2024-06-26T07:09:03.5924855Z 2060028f7488: Pull complete 2024-06-26T07:09:03.6980099Z e412c7f3b2de: Pull complete 2024-06-26T07:09:04.3005673Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:09:04.3147540Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:04.3211879Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:04.3263499Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:04.3264683Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:04.3272840Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:04.3273346Z env: 2024-06-26T07:09:04.3273630Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:04.3273984Z ##[endgroup] 2024-06-26T07:09:04.3429730Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:09:04.3430330Z with: 2024-06-26T07:09:04.3430637Z driver-version: 550.54.15 2024-06-26T07:09:04.3430993Z env: 2024-06-26T07:09:04.3431279Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:04.3431634Z ##[endgroup] 2024-06-26T07:09:04.3473113Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:09:04.3473691Z with: 2024-06-26T07:09:04.3473983Z timeout_minutes: 10 2024-06-26T07:09:04.3474315Z max_attempts: 3 2024-06-26T07:09:04.3505948Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:09:04.3537676Z retry_wait_seconds: 10 2024-06-26T07:09:04.3538064Z polling_interval_seconds: 1 2024-06-26T07:09:04.3538449Z warning_on_retry: true 2024-06-26T07:09:04.3538835Z continue_on_error: false 2024-06-26T07:09:04.3539189Z env: 2024-06-26T07:09:04.3539475Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:04.3539841Z DRIVER_VERSION: 550.54.15 2024-06-26T07:09:04.3540207Z ##[endgroup] 2024-06-26T07:09:04.4065910Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:09:04.4066961Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:09:04.4067485Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:09:04.7134671Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:04.7527220Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:09:04.7767321Z No Packages marked for removal 2024-06-26T07:09:04.7882171Z + install_nvidia_driver_common 2024-06-26T07:09:04.7883957Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:09:04.7884395Z + lspci 2024-06-26T07:09:04.7884702Z Before installing NVIDIA driver 2024-06-26T07:09:04.7962038Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:04.7962826Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:04.7963770Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:04.7964579Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:04.7965354Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:04.7966219Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:04.7966952Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:04.7967743Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:04.7968335Z + lsmod 2024-06-26T07:09:04.7976054Z Module Size Used by 2024-06-26T07:09:04.7976472Z xt_nat 16384 0 2024-06-26T07:09:04.7976855Z nvidia_modeset 1339392 0 2024-06-26T07:09:04.7977235Z veth 16384 0 2024-06-26T07:09:04.7977609Z nvidia_uvm 4599808 0 2024-06-26T07:09:04.7978075Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:04.7978726Z drm 421888 1 nvidia 2024-06-26T07:09:04.7979164Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:04.7979637Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:04.7980082Z xt_conntrack 16384 1 2024-06-26T07:09:04.7980459Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:04.7980904Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:04.7981380Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:04.7981832Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:04.7982311Z xfrm_user 45056 1 2024-06-26T07:09:04.7982715Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:04.7983140Z iptable_nat 16384 1 2024-06-26T07:09:04.7983516Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:04.7984162Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:04.7984693Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:04.7985250Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:04.7986265Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:04.7987166Z xt_addrtype 16384 2 2024-06-26T07:09:04.7987549Z iptable_filter 16384 1 2024-06-26T07:09:04.7987930Z br_netfilter 24576 0 2024-06-26T07:09:04.7988339Z bridge 172032 1 br_netfilter 2024-06-26T07:09:04.7988777Z stp 16384 1 bridge 2024-06-26T07:09:04.7989200Z llc 16384 2 bridge,stp 2024-06-26T07:09:04.7989616Z overlay 86016 0 2024-06-26T07:09:04.7989984Z sunrpc 393216 1 2024-06-26T07:09:04.7990353Z dm_mirror 28672 0 2024-06-26T07:09:04.7990748Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:04.7994096Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:04.7994894Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:04.7995514Z dax 69632 1 dm_mod 2024-06-26T07:09:04.7996054Z crc32_pclmul 16384 0 2024-06-26T07:09:04.7996821Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:04.7997274Z pcbc 16384 0 2024-06-26T07:09:04.7997639Z aesni_intel 188416 0 2024-06-26T07:09:04.7998027Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:04.7998555Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:04.7999137Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:04.7999698Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:04.8000243Z mousedev 24576 0 2024-06-26T07:09:04.8000610Z evdev 20480 3 2024-06-26T07:09:04.8000975Z psmouse 32768 0 2024-06-26T07:09:04.8001374Z button 16384 0 2024-06-26T07:09:04.8001731Z ena 139264 0 2024-06-26T07:09:04.8002093Z ptp 20480 1 ena 2024-06-26T07:09:04.8002498Z pps_core 20480 1 ptp 2024-06-26T07:09:04.8002894Z crc32c_intel 24576 0 2024-06-26T07:09:04.8003255Z autofs4 49152 2 2024-06-26T07:09:04.8003633Z + modinfo nvidia 2024-06-26T07:09:04.8004380Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:04.8005073Z alias: char-major-195-* 2024-06-26T07:09:04.8005444Z version: 550.54.15 2024-06-26T07:09:04.8005788Z supported: external 2024-06-26T07:09:04.8006231Z license: NVIDIA 2024-06-26T07:09:04.8006605Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:04.8007087Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:04.8007558Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:04.8008028Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:04.8008516Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:04.8009008Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:04.8009634Z depends: i2c-core,drm 2024-06-26T07:09:04.8009999Z retpoline: Y 2024-06-26T07:09:04.8010297Z name: nvidia 2024-06-26T07:09:04.8010853Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:04.8011511Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:04.8012159Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:04.8012750Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:04.8013196Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:04.8013627Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:04.8014083Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:04.8014516Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:04.8014956Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:04.8015484Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:04.8016055Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:04.8016542Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:04.8016969Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:04.8017394Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:04.8017842Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:04.8018370Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:04.8018955Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:04.8019515Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:04.8020116Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:04.8020716Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:04.8021325Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:04.8021921Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:04.8022405Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:04.8022949Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:04.8023504Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:04.8024246Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:04.8024845Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:04.8025327Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:04.8025800Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:04.8026247Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:04.8026742Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:04.8027273Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:04.8027752Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:04.8028230Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:04.8028729Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:04.8029213Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:04.8029712Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:04.8030188Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:04.8030597Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:04.8031064Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:04.8031541Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:04.8032000Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:04.8032480Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:04.8032959Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:04.8033408Z parm: rm_firmware_active:charp 2024-06-26T07:09:04.8033816Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:09:04.8034191Z ++ command -v nvidia-smi 2024-06-26T07:09:04.8034581Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:09:04.8034937Z + set +e 2024-06-26T07:09:04.8035435Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:09:06.7988678Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:09:06.7989326Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:06.7989813Z + '[' 0 -ne 0 ']' 2024-06-26T07:09:06.7990485Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:09:06.7990882Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:09:06.7991605Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:09:06.7992330Z + set -e 2024-06-26T07:09:06.7992644Z + '[' 1 -eq 0 ']' 2024-06-26T07:09:06.7993233Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:09:06.7993922Z + post_install_nvidia_driver_common 2024-06-26T07:09:06.7994342Z + sudo modprobe nvidia 2024-06-26T07:09:06.8089756Z + echo 'After installing NVIDIA driver' 2024-06-26T07:09:06.8090313Z + lspci 2024-06-26T07:09:06.8090674Z After installing NVIDIA driver 2024-06-26T07:09:06.8168061Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:06.8169100Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:06.8170073Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:06.8170869Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:06.8171635Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:06.8172408Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:06.8173138Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:06.8173919Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:06.8174521Z + lsmod 2024-06-26T07:09:06.8182294Z Module Size Used by 2024-06-26T07:09:06.8182709Z xt_nat 16384 0 2024-06-26T07:09:06.8183210Z nvidia_modeset 1339392 0 2024-06-26T07:09:06.8183723Z veth 16384 0 2024-06-26T07:09:06.8184617Z nvidia_uvm 4599808 0 2024-06-26T07:09:06.8185228Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:06.8185875Z drm 421888 1 nvidia 2024-06-26T07:09:06.8186345Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:06.8186817Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:06.8187431Z xt_conntrack 16384 1 2024-06-26T07:09:06.8187825Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:06.8188267Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:06.8188739Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:06.8189192Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:06.8189650Z xfrm_user 45056 1 2024-06-26T07:09:06.8190123Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:06.8190696Z iptable_nat 16384 1 2024-06-26T07:09:06.8191214Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:06.8191761Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:06.8192253Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:06.8192807Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:06.8193821Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:06.8194696Z xt_addrtype 16384 2 2024-06-26T07:09:06.8195130Z iptable_filter 16384 1 2024-06-26T07:09:06.8195515Z br_netfilter 24576 0 2024-06-26T07:09:06.8195926Z bridge 172032 1 br_netfilter 2024-06-26T07:09:06.8196366Z stp 16384 1 bridge 2024-06-26T07:09:06.8196787Z llc 16384 2 bridge,stp 2024-06-26T07:09:06.8197211Z overlay 86016 0 2024-06-26T07:09:06.8197587Z sunrpc 393216 1 2024-06-26T07:09:06.8197960Z dm_mirror 28672 0 2024-06-26T07:09:06.8198410Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:06.8198899Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:06.8199488Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:06.8199938Z dax 69632 1 dm_mod 2024-06-26T07:09:06.8200468Z crc32_pclmul 16384 0 2024-06-26T07:09:06.8200864Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:06.8201256Z pcbc 16384 0 2024-06-26T07:09:06.8201641Z aesni_intel 188416 0 2024-06-26T07:09:06.8202043Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:06.8202502Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:06.8202956Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:06.8203521Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:06.8204074Z mousedev 24576 0 2024-06-26T07:09:06.8204458Z evdev 20480 3 2024-06-26T07:09:06.8204824Z psmouse 32768 0 2024-06-26T07:09:06.8205196Z button 16384 0 2024-06-26T07:09:06.8205562Z ena 139264 0 2024-06-26T07:09:06.8206028Z ptp 20480 1 ena 2024-06-26T07:09:06.8206437Z pps_core 20480 1 ptp 2024-06-26T07:09:06.8206846Z crc32c_intel 24576 0 2024-06-26T07:09:06.8207226Z autofs4 49152 2 2024-06-26T07:09:06.8207594Z + modinfo nvidia 2024-06-26T07:09:06.8208273Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:06.8208987Z alias: char-major-195-* 2024-06-26T07:09:06.8209385Z version: 550.54.15 2024-06-26T07:09:06.8209728Z supported: external 2024-06-26T07:09:06.8210085Z license: NVIDIA 2024-06-26T07:09:06.8210483Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:06.8210975Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:06.8211439Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:06.8211920Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:06.8212441Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:06.8212940Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:06.8213413Z depends: i2c-core,drm 2024-06-26T07:09:06.8213795Z retpoline: Y 2024-06-26T07:09:06.8214126Z name: nvidia 2024-06-26T07:09:06.8214682Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:06.8215459Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:06.8216126Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:06.8216740Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:06.8217206Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:06.8217643Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:06.8218118Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:06.8218565Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:06.8219009Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:06.8219550Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:06.8220135Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:06.8220623Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:06.8221057Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:06.8221486Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:06.8221956Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:06.8222505Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:06.8223101Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:06.8223684Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:06.8224535Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:06.8225205Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:06.8225828Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:06.8226445Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:06.8226945Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:06.8227491Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:06.8228135Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:06.8228673Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:06.8229171Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:06.8229681Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:06.8230169Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:06.8230642Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:06.8231162Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:06.8231702Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:06.8232216Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:06.8232712Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:06.8233250Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:06.8233744Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:06.8234279Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:06.8234783Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:06.8235229Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:06.8235713Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:06.8236218Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:06.8236707Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:06.8237209Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:06.8237693Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:06.8238170Z parm: rm_firmware_active:charp 2024-06-26T07:09:06.8238586Z + set +e 2024-06-26T07:09:06.8238888Z + nvidia-smi 2024-06-26T07:09:08.3803701Z Wed Jun 26 07:09:08 2024 2024-06-26T07:09:08.3804514Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:08.3805366Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:09:08.3806286Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:08.3807092Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:09:08.3807997Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:09:08.3808930Z | | | MIG M. | 2024-06-26T07:09:08.3809477Z |=========================================+========================+======================| 2024-06-26T07:09:08.3910828Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:09:08.3911553Z | 0% 31C P0 57W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:09:08.3912190Z | | | N/A | 2024-06-26T07:09:08.3912858Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:08.3913570Z 2024-06-26T07:09:08.3914245Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:08.3914879Z | Processes: | 2024-06-26T07:09:08.3915606Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:09:08.3916305Z | ID ID Usage | 2024-06-26T07:09:08.3916885Z |=========================================================================================| 2024-06-26T07:09:08.3917537Z | No running processes found | 2024-06-26T07:09:08.3918291Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:09.0030256Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:09:10.5694680Z NVIDIA A10G 2024-06-26T07:09:11.0070617Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:11.0071198Z + '[' 0 -eq 0 ']' 2024-06-26T07:09:11.0071617Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:09:11.0072044Z + set -e 2024-06-26T07:09:11.0072360Z INFO: Ignoring allowed status 0 2024-06-26T07:09:11.0074211Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:09:11.0076515Z + sudo yum install -y yum-utils 2024-06-26T07:09:11.3129659Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:12.8120322Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:09:12.8121017Z Nothing to do 2024-06-26T07:09:12.9620173Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:09:12.9621161Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:12.9622553Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:13.2814984Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:13.3085796Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:13.3087113Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:13.3088078Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:13.3205214Z + sudo yum install -y nvidia-docker2 2024-06-26T07:09:13.6239928Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:15.0924066Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:09:15.0924714Z Nothing to do 2024-06-26T07:09:15.2444083Z + sudo systemctl restart docker 2024-06-26T07:09:53.4541289Z Command completed after 1 attempt(s). 2024-06-26T07:09:53.4599800Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:09:53.4600577Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:09:53.4601281Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:09:53.4601945Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:09:53.4609884Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:53.4610394Z env: 2024-06-26T07:09:53.4610691Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:53.4611175Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:53.4611670Z ##[endgroup] 2024-06-26T07:09:53.6838025Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:09:53.6998441Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:09:53.7098234Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:09:53.8174343Z Prepare all required actions 2024-06-26T07:09:53.8174854Z Getting action download info 2024-06-26T07:09:53.9257940Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:09:54.1558662Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:09:54.2617593Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:09:54.2618077Z with: 2024-06-26T07:09:54.2618419Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:54.2618883Z s3-bucket: gha-artifacts 2024-06-26T07:09:54.2619224Z env: 2024-06-26T07:09:54.2619497Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:54.2619965Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:54.2620452Z ##[endgroup] 2024-06-26T07:09:54.2652763Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:09:54.2653206Z with: 2024-06-26T07:09:54.2653587Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:54.2654057Z s3-bucket: gha-artifacts 2024-06-26T07:09:54.2654407Z region: us-east-1 2024-06-26T07:09:54.2654719Z env: 2024-06-26T07:09:54.2655001Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:54.2655456Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:54.2655959Z ##[endgroup] 2024-06-26T07:09:54.6904601Z (node:22609) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:09:54.6905335Z 2024-06-26T07:09:54.6905603Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:09:54.6906350Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:09:54.6907352Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:09:54.7944521Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:09:54.7945657Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:02.3075892Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:02.3082126Z Artifact download has finished successfully 2024-06-26T07:10:02.3229490Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:10:02.3229942Z unzip -o artifacts.zip 2024-06-26T07:10:02.3237778Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:02.3238293Z env: 2024-06-26T07:10:02.3238590Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:02.3239064Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:02.3239559Z ##[endgroup] 2024-06-26T07:10:02.3267048Z Archive: artifacts.zip 2024-06-26T07:10:02.3268704Z creating: dist/ 2024-06-26T07:10:04.3114969Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:04.3115660Z creating: build/custom_test_artifacts/ 2024-06-26T07:10:04.3116251Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:10:04.3117009Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:10:04.3117860Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:04.3121913Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:04.3123071Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:04.3124026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:04.3125022Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:04.3126098Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:04.3127211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:04.3128422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:04.3129425Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:04.3130451Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:04.3131641Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:04.3132810Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:04.3133959Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:04.3135105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:04.3136271Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:04.3137463Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:04.3138501Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:04.3139515Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:04.3173455Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:04.3211015Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:04.3212512Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:04.3254991Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:04.3256418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:04.3257855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:04.3259339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:04.3260779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:04.3262179Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:04.3263577Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:04.3265151Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:04.3266532Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:04.3267830Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:04.3269091Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:04.3270445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:04.3271704Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:04.3272918Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:04.3274151Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:04.3321646Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:04.3379388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:04.3380591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:04.3381653Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:04.3382526Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:04.3383431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:04.3384572Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:10:04.3385624Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:04.3386790Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:10:04.3387911Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:10:04.3388957Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:10:04.3390042Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:04.3391199Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:10:04.3392285Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:04.3393375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:10:04.3394443Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:10:04.3404022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:10:04.3531625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:10:04.3532649Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:10:04.3533755Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:04.3534978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:10:04.3536152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:10:04.3537263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:10:04.3538411Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:04.3539554Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:10:04.3540706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:04.3541903Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:10:04.3543035Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:10:04.3555344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:10:04.3634494Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:10:04.3635697Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:04.3636775Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:04.3637856Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:10:04.3638780Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:10:04.3639684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:04.3640597Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:10:04.3641470Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:10:04.3642270Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:10:04.3643082Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:10:04.3747694Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:10:04.3808792Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:10:04.3810291Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:10:04.3811210Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:10:04.3812041Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:04.3815401Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:04.3816460Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:04.3817416Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:04.3818681Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:04.3819916Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:04.3821081Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:04.3822442Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:04.3823667Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:04.3825166Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:04.3826464Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:04.3828018Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:04.3829167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:04.3830314Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:04.3831481Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:04.3832653Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:04.3833704Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:04.3834722Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:04.3866913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:04.3904795Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:04.3906261Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:04.3948438Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:04.3950086Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:04.3951777Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:04.3953479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:04.3955030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:04.3956526Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:04.3957940Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:04.3959347Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:04.3960715Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:04.3961999Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:04.3963250Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:04.3964479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:04.3965842Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:04.3967038Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:04.3968255Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:04.4014534Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:04.4072261Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:04.4073603Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:04.4074930Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:04.4076050Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:04.4077248Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:04.4078246Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:10:04.4079328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:10:04.4080537Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:10:04.4081721Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:10:04.4082800Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:10:04.4083927Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:10:04.4085170Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:10:04.4086398Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:10:04.4087511Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:10:04.4088615Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:10:04.4097621Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:10:04.4158801Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:10:04.4160153Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:04.4161422Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:04.4162409Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:10:04.4163395Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:10:04.4164337Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:04.4165316Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:10:04.4166264Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:10:04.4167044Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:10:04.4167857Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:10:04.4212830Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:10:04.4213762Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:10:04.4214531Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:10:04.4215434Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:04.4220658Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:04.4221886Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:04.4222897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:04.4224457Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:04.4225802Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:04.4226975Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:04.4228408Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:04.4229705Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:04.4231098Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:04.4232479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:04.4234044Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:04.4235250Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:04.4236465Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:04.4237708Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:04.4238938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:04.4240168Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:04.4241256Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:04.4272743Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:04.4310735Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:04.4312381Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:04.4354696Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:04.4356469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:04.4357979Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:04.4359525Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:04.4361008Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:04.4378945Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:04.4380463Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:04.4381936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:04.4404522Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:04.4406025Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:04.4407335Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:04.4408619Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:04.4409913Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:04.4411209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:04.4412492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:04.4442885Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:04.4501083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:04.4502330Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:04.4503372Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:04.4504528Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:04.4505497Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:04.4506504Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:10:04.4507641Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:04.4509058Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:10:04.4510275Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:10:04.4511470Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:10:04.4512640Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:04.4513914Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:10:04.4515105Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:04.4516296Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:10:04.4517492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:10:04.4518746Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:10:04.4630989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:10:04.4632167Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:10:04.4633356Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:04.4634682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:10:04.4635947Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:10:04.4637146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:10:04.4638398Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:04.4639648Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:10:04.4640901Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:04.4642141Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:10:04.4643372Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:10:04.4654803Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:10:04.4707514Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:10:04.4708827Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:04.4709974Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:04.4711029Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:10:04.4712013Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:10:04.4712989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:04.4713967Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:10:04.4714879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:10:04.4715726Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:10:04.4716708Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:10:04.4812601Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:10:04.4853984Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:10:04.4854623Z creating: build/lib/ 2024-06-26T07:10:04.4939399Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:10:04.5376818Z inflating: build/lib/libprotobuf.a 2024-06-26T07:10:04.5386297Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:10:04.5393960Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:10:04.5401505Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:10:04.5402387Z inflating: build/lib/libclog.a 2024-06-26T07:10:04.5419513Z inflating: build/lib/libnnpack.a 2024-06-26T07:10:04.5421488Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:10:04.5483858Z inflating: build/lib/libgtest.a 2024-06-26T07:10:04.5556223Z inflating: build/lib/libbenchmark.a 2024-06-26T07:10:04.5617919Z inflating: build/lib/libasmjit.a 2024-06-26T07:10:04.5625038Z inflating: build/lib/libittnotify.a 2024-06-26T07:10:04.5651648Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:10:04.5773362Z inflating: build/lib/libgloo.a 2024-06-26T07:10:04.5794264Z inflating: build/lib/libfmt.a 2024-06-26T07:10:04.5884068Z inflating: build/lib/libc10.so 2024-06-26T07:10:04.5885542Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:10:04.5886144Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:10:04.5887367Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:10:04.5906268Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:10:04.5923344Z inflating: build/lib/libgmock.a 2024-06-26T07:10:04.5923790Z inflating: build/lib/libgtest_main.a 2024-06-26T07:10:04.5924523Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:10:04.6411003Z inflating: build/lib/libprotoc.a 2024-06-26T07:10:04.6779767Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:10:05.6508174Z inflating: build/lib/libdnnl.a 2024-06-26T07:10:05.7057609Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:10:05.7110497Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:10:05.7110956Z inflating: build/lib/libgmock_main.a 2024-06-26T07:10:05.8310667Z inflating: build/lib/libfbgemm.a 2024-06-26T07:10:05.8556050Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:10:05.9052553Z inflating: build/lib/libkineto.a 2024-06-26T07:10:05.9230286Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:10:05.9271359Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:10:05.9934999Z inflating: build/lib/libonnx.a 2024-06-26T07:10:08.3187614Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:10:08.3191557Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:10:08.3195358Z inflating: build/lib/libshm.so 2024-06-26T07:10:10.3034398Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:10:10.3035195Z inflating: build/lib/libtorch.so 2024-06-26T07:10:10.3038062Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:10:11.0977476Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:10:11.2869428Z inflating: build/lib/libtorch_python.so 2024-06-26T07:10:11.2935679Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:10:11.2955721Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:10:11.2980280Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:10:11.3013758Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:10:11.3014235Z creating: build/bin/ 2024-06-26T07:10:11.3062699Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:10:11.3112841Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:10:11.3161683Z inflating: build/bin/c10_Device_test 2024-06-26T07:10:11.3219090Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:10:11.3271155Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:10:11.3318696Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:10:11.3368145Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:10:11.3421120Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:10:11.3475097Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:10:11.3529980Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:10:11.3598852Z inflating: build/bin/c10_cow_test 2024-06-26T07:10:11.3650459Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:10:11.3698019Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:10:11.3746646Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:10:11.3796234Z inflating: build/bin/c10_Half_test 2024-06-26T07:10:11.3851291Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:10:11.3904958Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:10:11.3953249Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:10:11.4007793Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:10:11.4057128Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:10:11.4106723Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:10:11.4154174Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:10:11.4204394Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:10:11.4258149Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:10:11.4307157Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:10:11.4361739Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:10:11.4415736Z inflating: build/bin/c10_complex_test 2024-06-26T07:10:11.4467510Z inflating: build/bin/c10_exception_test 2024-06-26T07:10:11.4515906Z inflating: build/bin/c10_flags_test 2024-06-26T07:10:11.4564817Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:10:11.4723171Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:10:11.4772353Z inflating: build/bin/c10_irange_test 2024-06-26T07:10:11.4824529Z inflating: build/bin/c10_lazy_test 2024-06-26T07:10:11.4879675Z inflating: build/bin/c10_logging_test 2024-06-26T07:10:11.4952657Z inflating: build/bin/c10_optional_test 2024-06-26T07:10:11.5013056Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:10:11.5065451Z inflating: build/bin/c10_registry_test 2024-06-26T07:10:11.5211168Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:10:11.5260866Z inflating: build/bin/c10_ssize_test 2024-06-26T07:10:11.5311390Z inflating: build/bin/c10_string_util_test 2024-06-26T07:10:11.5368197Z inflating: build/bin/c10_string_view_test 2024-06-26T07:10:11.5416987Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:10:11.5464578Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:10:11.5518816Z inflating: build/bin/c10_typeid_test 2024-06-26T07:10:11.5951761Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:10:11.6385313Z inflating: build/bin/protoc 2024-06-26T07:10:11.6436583Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:10:11.6487849Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:10:11.6539109Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:10:11.6589857Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:10:11.6640978Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:10:11.6692206Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:10:11.6739757Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:10:11.6791691Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:10:11.7109905Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:10:11.7441588Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:10:11.7786871Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:10:11.7838282Z inflating: build/bin/FileStoreTest 2024-06-26T07:10:11.7892394Z inflating: build/bin/TCPStoreTest 2024-06-26T07:10:11.7945114Z inflating: build/bin/HashStoreTest 2024-06-26T07:10:11.7957326Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:10:11.8010214Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:10:11.8013755Z inflating: build/bin/torch_shm_manager 2024-06-26T07:10:11.8016546Z inflating: build/bin/example_allreduce 2024-06-26T07:10:11.8071262Z inflating: build/bin/test_dist_autograd 2024-06-26T07:10:11.8138865Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:10:11.8141131Z inflating: build/bin/parallel_benchmark 2024-06-26T07:10:11.8206304Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:10:11.8215118Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:10:11.8547904Z inflating: build/bin/test_lazy 2024-06-26T07:10:11.9695510Z inflating: build/bin/test_api 2024-06-26T07:10:11.9767068Z inflating: build/bin/Dict_test 2024-06-26T07:10:11.9817523Z inflating: build/bin/Dimname_test 2024-06-26T07:10:11.9880160Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:10:11.9935797Z inflating: build/bin/NamedTensor_test 2024-06-26T07:10:11.9993277Z inflating: build/bin/apply_utils_test 2024-06-26T07:10:12.0050503Z inflating: build/bin/atest 2024-06-26T07:10:12.0111769Z inflating: build/bin/basic 2024-06-26T07:10:12.0165840Z inflating: build/bin/broadcast_test 2024-06-26T07:10:12.0214240Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:10:12.0270740Z inflating: build/bin/cpu_generator_test 2024-06-26T07:10:12.0322346Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:10:12.0412754Z inflating: build/bin/cpu_rng_test 2024-06-26T07:10:12.0461204Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:10:12.0510511Z inflating: build/bin/dlconvertor_test 2024-06-26T07:10:12.0566362Z inflating: build/bin/extension_backend_test 2024-06-26T07:10:12.0620252Z inflating: build/bin/half_test 2024-06-26T07:10:12.0713423Z inflating: build/bin/ivalue_test 2024-06-26T07:10:12.0761347Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:10:12.0814054Z inflating: build/bin/math_kernel_test 2024-06-26T07:10:12.0867224Z inflating: build/bin/memory_format_test 2024-06-26T07:10:12.0917951Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:10:12.0969705Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:10:12.1023415Z inflating: build/bin/native_test 2024-06-26T07:10:12.1072657Z inflating: build/bin/operator_name_test 2024-06-26T07:10:12.1122574Z inflating: build/bin/operators_test 2024-06-26T07:10:12.1172622Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:10:12.1237773Z inflating: build/bin/pow_test 2024-06-26T07:10:12.1293237Z inflating: build/bin/quantized_test 2024-06-26T07:10:12.1341671Z inflating: build/bin/reduce_ops_test 2024-06-26T07:10:12.1391239Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:10:12.1446004Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:10:12.1502665Z inflating: build/bin/scalar_test 2024-06-26T07:10:12.1553297Z inflating: build/bin/StorageUtils_test 2024-06-26T07:10:12.1603743Z inflating: build/bin/stride_properties_test 2024-06-26T07:10:12.1679998Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:10:12.1732546Z inflating: build/bin/test_parallel 2024-06-26T07:10:12.1735337Z inflating: build/bin/thread_init_test 2024-06-26T07:10:12.1789562Z inflating: build/bin/type_ptr_test 2024-06-26T07:10:12.1847931Z inflating: build/bin/type_test 2024-06-26T07:10:12.1898041Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:10:12.1899407Z inflating: build/bin/verify_api_visibility 2024-06-26T07:10:12.1966758Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:10:12.2016803Z inflating: build/bin/weakref_test 2024-06-26T07:10:12.2066735Z inflating: build/bin/wrapdim_test 2024-06-26T07:10:12.2117136Z inflating: build/bin/xla_tensor_test 2024-06-26T07:10:12.2175189Z inflating: build/bin/IListRef_test 2024-06-26T07:10:12.2278054Z inflating: build/bin/List_test 2024-06-26T07:10:12.2342584Z inflating: build/bin/KernelFunction_test 2024-06-26T07:10:12.2460373Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:10:12.2553963Z inflating: build/bin/kernel_function_test 2024-06-26T07:10:12.2678137Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:10:12.2778111Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:10:12.2837307Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:10:12.2930451Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:10:12.2979790Z inflating: build/bin/CppSignature_test 2024-06-26T07:10:12.3034096Z inflating: build/bin/backend_fallback_test 2024-06-26T07:10:12.3081312Z inflating: build/bin/op_allowlist_test 2024-06-26T07:10:12.3142955Z inflating: build/bin/inline_container_test 2024-06-26T07:10:12.3435744Z inflating: build/bin/op_registration_test 2024-06-26T07:10:12.3487065Z inflating: build/bin/cuda_apply_test 2024-06-26T07:10:12.3536333Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:10:12.3594077Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:10:12.3646184Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:10:12.3713782Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:10:12.3770998Z inflating: build/bin/cuda_complex_test 2024-06-26T07:10:12.3819560Z inflating: build/bin/cuda_device_test 2024-06-26T07:10:12.3875099Z inflating: build/bin/cuda_cub_test 2024-06-26T07:10:12.3924303Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:10:12.3987021Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:10:12.4042002Z inflating: build/bin/cuda_generator_test 2024-06-26T07:10:12.4090787Z inflating: build/bin/cuda_half_test 2024-06-26T07:10:12.4140229Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:10:12.4187870Z inflating: build/bin/cuda_optional_test 2024-06-26T07:10:12.4237782Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:10:12.4289255Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:10:12.4337801Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:10:12.4396711Z inflating: build/bin/cuda_stream_test 2024-06-26T07:10:12.4444960Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:10:12.4494791Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:10:12.4509247Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:10:12.4572867Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:10:12.4628504Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:10:12.4689808Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:10:12.4750232Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:10:12.5556272Z inflating: build/bin/test_tensorexpr 2024-06-26T07:10:12.6116118Z inflating: build/bin/test_jit 2024-06-26T07:10:12.6116554Z creating: .additional_ci_files/ 2024-06-26T07:10:12.6170283Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:10:12.6381041Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:10:12.6407869Z ##[group]Run rm artifacts.zip 2024-06-26T07:10:12.6408254Z rm artifacts.zip 2024-06-26T07:10:12.6415815Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:12.6416321Z env: 2024-06-26T07:10:12.6416617Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:12.6417070Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:12.6417560Z ##[endgroup] 2024-06-26T07:10:12.7125300Z ##[group]Run df -H 2024-06-26T07:10:12.7125626Z df -H 2024-06-26T07:10:12.7133393Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:12.7134036Z env: 2024-06-26T07:10:12.7134326Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:12.7134790Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:12.7135291Z ##[endgroup] 2024-06-26T07:10:12.7161880Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:10:12.7162567Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:10:12.7163420Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:10:12.7163896Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:10:12.7164376Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:10:12.7164877Z /dev/nvme0n1p1 162G 44G 118G 27% / 2024-06-26T07:10:12.7202638Z Prepare all required actions 2024-06-26T07:10:12.7203082Z Getting action download info 2024-06-26T07:10:12.8760293Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:10:12.8760761Z with: 2024-06-26T07:10:12.8761029Z env: 2024-06-26T07:10:12.8761314Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:12.8761782Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:12.8762275Z ##[endgroup] 2024-06-26T07:10:12.8793165Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:12.8793620Z with: 2024-06-26T07:10:12.8793889Z name: td_results 2024-06-26T07:10:12.8794216Z s3-bucket: gha-artifacts 2024-06-26T07:10:12.8794579Z region: us-east-1 2024-06-26T07:10:12.8794882Z env: 2024-06-26T07:10:12.8795153Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:12.8795618Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:12.8796110Z ##[endgroup] 2024-06-26T07:10:13.2855422Z (node:22637) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:13.2856255Z 2024-06-26T07:10:13.2856535Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:13.2857263Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:13.2858105Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:13.3501219Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:10:13.3508443Z Artifact download has finished successfully 2024-06-26T07:10:13.3636958Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:10:13.3637449Z mkdir -p .additional_ci_files 2024-06-26T07:10:13.3638017Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:10:13.3646154Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:13.3646648Z env: 2024-06-26T07:10:13.3646929Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:13.3647399Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:13.3647888Z ##[endgroup] 2024-06-26T07:10:13.3681973Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:10:13.3694033Z ##[error]Process completed with exit code 1. 2024-06-26T07:10:13.3739710Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:10:13.3740199Z .github/scripts/parse_ref.py 2024-06-26T07:10:13.3747427Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:13.3747785Z env: 2024-06-26T07:10:13.3748079Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:13.3748555Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:13.3749054Z ##[endgroup] 2024-06-26T07:10:13.3976545Z Prepare all required actions 2024-06-26T07:10:13.4013796Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:10:13.4014265Z with: 2024-06-26T07:10:13.4014821Z github-token: *** 2024-06-26T07:10:13.4015149Z env: 2024-06-26T07:10:13.4015432Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:13.4015907Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:13.4016411Z ##[endgroup] 2024-06-26T07:10:13.4032285Z ##[group]Run set -eux 2024-06-26T07:10:13.4032641Z set -eux 2024-06-26T07:10:13.4033306Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:10:13.4041200Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:13.4041867Z env: 2024-06-26T07:10:13.4042169Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:13.4042661Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:13.4043339Z GITHUB_TOKEN: *** 2024-06-26T07:10:13.4043672Z ##[endgroup] 2024-06-26T07:10:13.4063170Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-02e64d1cff9e6c2ad 2024-06-26T07:10:14.6459045Z setting job-id=26688911970 2024-06-26T07:10:14.6459965Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:14.6655369Z Prepare all required actions 2024-06-26T07:10:14.6655824Z Getting action download info 2024-06-26T07:10:14.7773377Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:10:14.7773846Z with: 2024-06-26T07:10:14.7774304Z github-token: *** 2024-06-26T07:10:14.7782553Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:10:14.7791593Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:14.7792347Z env: 2024-06-26T07:10:14.7792632Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:14.7793095Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:14.7793580Z ##[endgroup] 2024-06-26T07:10:14.7834393Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:10:14.7834963Z with: 2024-06-26T07:10:14.7835258Z shell: bash 2024-06-26T07:10:14.7835579Z timeout_minutes: 10 2024-06-26T07:10:14.7835929Z max_attempts: 5 2024-06-26T07:10:14.7836247Z retry_wait_seconds: 30 2024-06-26T07:10:14.7837399Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:14.7838597Z polling_interval_seconds: 1 2024-06-26T07:10:14.7838997Z warning_on_retry: true 2024-06-26T07:10:14.7839362Z continue_on_error: false 2024-06-26T07:10:14.7839728Z env: 2024-06-26T07:10:14.7840023Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:14.7840631Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:14.7841296Z GITHUB_TOKEN: *** 2024-06-26T07:10:14.7841632Z ##[endgroup] 2024-06-26T07:10:14.8334106Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:15.0499154Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:15.0651788Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:10:15.0794309Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:10:15.0803075Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:10:15.0823275Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:10:15.0832215Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:10:15.1029893Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:10:15.8330182Z Command completed after 1 attempt(s). 2024-06-26T07:10:15.8370954Z ##[group]Run set -x 2024-06-26T07:10:15.8371303Z set -x 2024-06-26T07:10:15.8371620Z  2024-06-26T07:10:15.8372207Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:15.8372901Z # in runner workspace 2024-06-26T07:10:15.8373451Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:10:15.8381276Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:15.8381799Z env: 2024-06-26T07:10:15.8382082Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:15.8382571Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:15.8383082Z ##[endgroup] 2024-06-26T07:10:15.8404056Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:10:15.8602154Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:15.8602716Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:15.8603186Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:10:15.8603600Z  2024-06-26T07:10:15.8604175Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:15.8605106Z # in runner workspace 2024-06-26T07:10:15.8605766Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:10:15.8606416Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:10:15.8606874Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:10:15.8615307Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:10:15.8624176Z  --selected-test-configs "" \ 2024-06-26T07:10:15.8624648Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:10:15.8625078Z  --tag "${TAG}" \ 2024-06-26T07:10:15.8625488Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:10:15.8625945Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:10:15.8626385Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:10:15.8633755Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:15.8634265Z env: 2024-06-26T07:10:15.8634546Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:15.8635020Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:15.8635705Z GITHUB_TOKEN: *** 2024-06-26T07:10:15.8636405Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:15.8637170Z PR_NUMBER: 2024-06-26T07:10:15.8637517Z TAG: ciflow/inductor/129470 2024-06-26T07:10:15.8637911Z EVENT_NAME: push 2024-06-26T07:10:15.8638247Z SCHEDULE: 2024-06-26T07:10:15.8638548Z HEAD_BRANCH: 2024-06-26T07:10:15.8638860Z ##[endgroup] 2024-06-26T07:10:15.8670777Z Workflow: inductor 2024-06-26T07:10:15.8671686Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:16.0790769Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:10:16.2382657Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:10:16.2383133Z echo "Filtered matrix:" 2024-06-26T07:10:16.2391662Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:10:16.2400439Z  2024-06-26T07:10:16.2400719Z echo 2024-06-26T07:10:16.2401114Z echo "Is the current job unstable? False" 2024-06-26T07:10:16.2401582Z  2024-06-26T07:10:16.2401862Z echo 2024-06-26T07:10:16.2402235Z echo "Is keep-going label set? False" 2024-06-26T07:10:16.2402691Z  2024-06-26T07:10:16.2402964Z echo 2024-06-26T07:10:16.2403292Z echo "Renabled issues? " 2024-06-26T07:10:16.2411117Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:16.2411624Z env: 2024-06-26T07:10:16.2411905Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:16.2412384Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:16.2412879Z ##[endgroup] 2024-06-26T07:10:16.2435443Z Filtered matrix: 2024-06-26T07:10:16.2445441Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:10:16.2453807Z 2024-06-26T07:10:16.2453970Z Is the current job unstable? False 2024-06-26T07:10:16.2454258Z 2024-06-26T07:10:16.2454499Z Is keep-going label set? False 2024-06-26T07:10:16.2454773Z 2024-06-26T07:10:16.2454902Z Renabled issues? 2024-06-26T07:10:16.2497756Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:16.2498517Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:16.2506389Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:16.2506892Z env: 2024-06-26T07:10:16.2507187Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:16.2507670Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:16.2508176Z JOB_TIMEOUT: 240 2024-06-26T07:10:16.2508491Z ##[endgroup] 2024-06-26T07:10:16.2580543Z ##[group]Run set -x 2024-06-26T07:10:16.2581058Z set -x 2024-06-26T07:10:16.2581368Z  2024-06-26T07:10:16.2581736Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:10:16.2582392Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:10:16.2582971Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:10:16.2583501Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:10:16.2584218Z else 2024-06-26T07:10:16.2584570Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:16.2585004Z fi 2024-06-26T07:10:16.2585298Z  2024-06-26T07:10:16.2585794Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:10:16.2586568Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:10:16.2587266Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:10:16.2587879Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:10:16.2588350Z container_name=$(docker run \ 2024-06-26T07:10:16.2588796Z  ${GPU_FLAG:-} \ 2024-06-26T07:10:16.2589184Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:10:16.2589597Z  -e PR_NUMBER \ 2024-06-26T07:10:16.2589985Z  -e GITHUB_ACTIONS \ 2024-06-26T07:10:16.2590390Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:10:16.2590812Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:10:16.2591213Z  -e GITHUB_JOB \ 2024-06-26T07:10:16.2591581Z  -e GITHUB_RUN_ID \ 2024-06-26T07:10:16.2591988Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:10:16.2592417Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:10:16.2592821Z  -e JOB_ID \ 2024-06-26T07:10:16.2593174Z  -e JOB_NAME \ 2024-06-26T07:10:16.2593540Z  -e BASE_SHA \ 2024-06-26T07:10:16.2593899Z  -e BRANCH \ 2024-06-26T07:10:16.2594243Z  -e SHA1 \ 2024-06-26T07:10:16.2594604Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:10:16.2595026Z  -e IN_WHEEL_TEST \ 2024-06-26T07:10:16.2595427Z  -e SHARD_NUMBER \ 2024-06-26T07:10:16.2595812Z  -e TEST_CONFIG \ 2024-06-26T07:10:16.2596198Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:10:16.2596600Z  -e REENABLED_ISSUES \ 2024-06-26T07:10:16.2597033Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:10:16.2597470Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:10:16.2597886Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:10:16.2598287Z  -e NO_TD \ 2024-06-26T07:10:16.2598640Z  -e TD_DISTRIBUTED \ 2024-06-26T07:10:16.2599027Z  -e PR_LABELS \ 2024-06-26T07:10:16.2599447Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:10:16.2599927Z  -e SCCACHE_BUCKET \ 2024-06-26T07:10:16.2600331Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:10:16.2600762Z  -e XLA_CUDA \ 2024-06-26T07:10:16.2601192Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:10:16.2601721Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:10:16.2602259Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:10:16.2602790Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:10:16.2603279Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:10:16.2603728Z  -e DASHBOARD_TAG \ 2024-06-26T07:10:16.2604215Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:10:16.2604799Z  --security-opt seccomp=unconfined \ 2024-06-26T07:10:16.2605284Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:10:16.2605775Z  --ipc=host \ 2024-06-26T07:10:16.2606146Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:10:16.2606563Z  --tty \ 2024-06-26T07:10:16.2606893Z  --detach \ 2024-06-26T07:10:16.2607273Z  --name="${container_name}" \ 2024-06-26T07:10:16.2607702Z  --user jenkins \ 2024-06-26T07:10:16.2608320Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:10:16.2608973Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:10:16.2609439Z  "${DOCKER_IMAGE}" 2024-06-26T07:10:16.2609806Z ) 2024-06-26T07:10:16.2610246Z # Propagate download.pytorch.org IP to container 2024-06-26T07:10:16.2611196Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:10:16.2612205Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:10:16.2613143Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:10:16.2620822Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:16.2621183Z env: 2024-06-26T07:10:16.2621480Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:16.2621953Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:16.2622580Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:16.2623103Z PR_NUMBER: 2024-06-26T07:10:16.2623456Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:10:16.2624083Z GITHUB_WORKFLOW: inductor 2024-06-26T07:10:16.2624486Z GITHUB_JOB: test 2024-06-26T07:10:16.2624835Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:10:16.2625212Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:10:16.2625596Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:10:16.2625943Z JOB_ID: 26688911970 2024-06-26T07:10:16.2626660Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:16.2627427Z BRANCH: 2024-06-26T07:10:16.2627785Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:16.2628309Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:16.2628834Z TEST_CONFIG: dynamic_inductor_torchbench 2024-06-26T07:10:16.2629293Z SHARD_NUMBER: 1 2024-06-26T07:10:16.2629631Z NUM_TEST_SHARDS: 2 2024-06-26T07:10:16.2629966Z REENABLED_ISSUES: 2024-06-26T07:10:16.2630332Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:10:16.2630756Z VERBOSE_TEST_LOGS: False 2024-06-26T07:10:16.2631135Z NO_TEST_TIMEOUT: False 2024-06-26T07:10:16.2631507Z NO_TD: False 2024-06-26T07:10:16.2631834Z TD_DISTRIBUTED: False 2024-06-26T07:10:16.2632282Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:10:16.2632801Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:10:16.2633209Z SHM_SIZE: 2g 2024-06-26T07:10:16.2634316Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:16.2635507Z XLA_CUDA: 2024-06-26T07:10:16.2636003Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:16.2636647Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:10:16.2637115Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:10:16.2637554Z DASHBOARD_TAG: 2024-06-26T07:10:16.2638101Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:10:16.2638485Z ##[endgroup] 2024-06-26T07:10:16.2658091Z + [[ dynamic_inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:10:16.2658826Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:10:16.2659351Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:16.2665329Z +++ nproc --ignore=2 2024-06-26T07:10:16.2677772Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:27.5379148Z + container_name=41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T07:10:27.5380117Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:10:27.5382783Z + docker exec -i 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:10:27.5981092Z + echo DOCKER_CONTAINER_ID=41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T07:10:27.5983791Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:27.5986105Z + docker exec -t 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:10:27.9723237Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:10:28.2855480Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:10:28.2859974Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:10:28.2862429Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:10:28.2864714Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:10:28.2868104Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:10:28.2870033Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:10:28.2896814Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:10:28.2956211Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:10:28.3394337Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:10:28.3553663Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:10:29.6760703Z Installing collected packages: torch 2024-06-26T07:10:38.4565278Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:10:38.4566977Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:10:38.4567910Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:10:38.5447061Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:10:38.5450726Z + source .ci/pytorch/common.sh 2024-06-26T07:10:38.5453156Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:10:38.5459433Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:10:38.5460097Z +++ declare -f -t trap_add 2024-06-26T07:10:38.5465705Z ++ set -ex 2024-06-26T07:10:38.5466434Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:10:38.5467021Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:10:38.5468621Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:10:38.5470203Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:10:38.5479186Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:10:38.5479705Z + trap_add cleanup_workspace EXIT 2024-06-26T07:10:38.5480121Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:10:38.5480493Z + shift 2024-06-26T07:10:38.5480771Z + for trap_add_name in "$@" 2024-06-26T07:10:38.5484773Z +++ trap -p EXIT 2024-06-26T07:10:38.5487188Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:10:38.5487631Z +++ extract_trap_cmd 2024-06-26T07:10:38.5487988Z +++ printf '%s\n' '' 2024-06-26T07:10:38.5488435Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:10:38.5489107Z + trap -- ' 2024-06-26T07:10:38.5489555Z cleanup_workspace' EXIT 2024-06-26T07:10:38.5490042Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:10:39.1899100Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:10:39.1911877Z + echo 'Environment variables:' 2024-06-26T07:10:39.1912400Z Environment variables: 2024-06-26T07:10:39.1912838Z + env 2024-06-26T07:10:39.1918078Z INSTALLED_DB=yes 2024-06-26T07:10:39.1918485Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:10:39.1918945Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:10:39.1919489Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:10:39.1920298Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:10:39.1921085Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:10:39.1921784Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:10:39.1922562Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:39.1923324Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:39.1923832Z HOSTNAME=41defb39909c 2024-06-26T07:10:39.1924783Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.1926060Z GITHUB_ACTION=__self 2024-06-26T07:10:39.1926537Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:10:39.1930727Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:10:39.1934619Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:10:39.1935134Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:10:39.1935493Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:10:39.1935863Z TEST_CONFIG=dynamic_inductor_torchbench 2024-06-26T07:10:39.1936304Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:10:39.1936775Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:10:39.1937242Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:10:39.1937677Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:10:39.1938110Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:10:39.1938556Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:39.1939052Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:10:39.1939523Z GITHUB_REF_TYPE=tag 2024-06-26T07:10:39.1939858Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:10:39.1940238Z NCCL_VERSION=2.17.1-1 2024-06-26T07:10:39.1940621Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.1941260Z XLA_CUDA= 2024-06-26T07:10:39.1941862Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:10:39.1944520Z *** 2024-06-26T07:10:39.1944827Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:10:39.1945252Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:10:39.1945629Z GITHUB_ACTIONS=true 2024-06-26T07:10:39.1945972Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:39.1946489Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:10:39.1947026Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:10:39.1947491Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.1947992Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:10:39.1948471Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.1949267Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:10:39.1949988Z UCC_HOME=/usr 2024-06-26T07:10:39.1950329Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:10:39.1950725Z VERBOSE_TEST_LOGS=False 2024-06-26T07:10:39.1951087Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:10:39.1951554Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:10:39.1952047Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:10:39.1952514Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:10:39.1952894Z SHARD_NUMBER=1 2024-06-26T07:10:39.1953205Z GITHUB_REF_PROTECTED=false 2024-06-26T07:10:39.1953553Z HOME=/var/lib/jenkins 2024-06-26T07:10:39.1953926Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:10:39.1954368Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:10:39.1954846Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:10:39.1955323Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:10:39.1955694Z CUDA_VERSION=12.1.1 2024-06-26T07:10:39.1956115Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:10:39.1956559Z NUM_TEST_SHARDS=2 2024-06-26T07:10:39.1956854Z UCX_HOME=/usr 2024-06-26T07:10:39.1957369Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:10:39.1958457Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.1959727Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:39.1960989Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.1962112Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:10:39.1962776Z GITHUB_EVENT_NAME=push 2024-06-26T07:10:39.1963110Z DASHBOARD_TAG= 2024-06-26T07:10:39.1963410Z GITHUB_RUN_ID=9673646808 2024-06-26T07:10:39.1963865Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:10:39.1964400Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:10:39.1965401Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.1966408Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:39.1966820Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:10:39.1967205Z PR_NUMBER= 2024-06-26T07:10:39.1967499Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:10:39.1967835Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:10:39.1968283Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:10:39.1968749Z TERM=xterm 2024-06-26T07:10:39.1969087Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:10:39.1969505Z INSTALLED_VISION=yes 2024-06-26T07:10:39.1969833Z BRANCH= 2024-06-26T07:10:39.1970118Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:10:39.1970563Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:10:39.1970992Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:10:39.1971746Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:10:39.1972498Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:10:39.1972981Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:10:39.1973449Z REENABLED_ISSUES= 2024-06-26T07:10:39.1973898Z SHLVL=1 2024-06-26T07:10:39.1974234Z MAX_JOBS=14 2024-06-26T07:10:39.1974551Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:10:39.1974907Z NVARCH=x86_64 2024-06-26T07:10:39.1975218Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:10:39.1975667Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.1976199Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:10:39.1976687Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:10:39.1977348Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:39.1977914Z GITHUB_JOB=test 2024-06-26T07:10:39.1978313Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:10:39.1978876Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:10:39.1979389Z NO_TEST_TIMEOUT=False 2024-06-26T07:10:39.1979713Z TD_DISTRIBUTED=False 2024-06-26T07:10:39.1980092Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:10:39.1980541Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:10:39.1980958Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:10:39.1981331Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:10:39.1981690Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:10:39.1982045Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:10:39.1983098Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:39.1984483Z GITHUB_BASE_REF= 2024-06-26T07:10:39.1984827Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:10:39.1985202Z CI=true 2024-06-26T07:10:39.1985516Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:39.1985936Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:10:39.1986329Z JOB_ID=26688911970 2024-06-26T07:10:39.1986649Z INSTALLED_PROTOBUF=yes 2024-06-26T07:10:39.1986984Z GITHUB_HEAD_REF= 2024-06-26T07:10:39.1987299Z GITHUB_ACTION_REF= 2024-06-26T07:10:39.1987725Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:10:39.1988190Z GITHUB_WORKFLOW=inductor 2024-06-26T07:10:39.1988562Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:10:39.1989471Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.1990301Z NO_TD=False 2024-06-26T07:10:39.1990603Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:10:39.1990979Z _=/usr/bin/env 2024-06-26T07:10:39.1991462Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:10:39.2064503Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:10:39.2065644Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:10:39.2066525Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:10:39.2067412Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:10:39.2068031Z + BUILD_DIR=build 2024-06-26T07:10:39.2068363Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:10:39.2068769Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:10:39.2069130Z + SHARD_NUMBER=1 2024-06-26T07:10:39.2069450Z + NUM_TEST_SHARDS=2 2024-06-26T07:10:39.2069775Z + export VALGRIND=ON 2024-06-26T07:10:39.2070103Z + VALGRIND=ON 2024-06-26T07:10:39.2070572Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:10:39.2071074Z + [[ 0 == \1 ]] 2024-06-26T07:10:39.2071372Z + [[ False == \1 ]] 2024-06-26T07:10:39.2071851Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:10:39.2072396Z ++ realpath build/custom_test_artifacts 2024-06-26T07:10:39.2075621Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:10:39.2076382Z + [[ -n '' ]] 2024-06-26T07:10:39.2076738Z + echo 'Environment variables' 2024-06-26T07:10:39.2077124Z Environment variables 2024-06-26T07:10:39.2077437Z + env 2024-06-26T07:10:39.2081353Z INSTALLED_DB=yes 2024-06-26T07:10:39.2081848Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:10:39.2082364Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:10:39.2083273Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:10:39.2083910Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:10:39.2084582Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:10:39.2085120Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:10:39.2085822Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:39.2086434Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:39.2087058Z HOSTNAME=41defb39909c 2024-06-26T07:10:39.2088072Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.2088998Z GITHUB_ACTION=__self 2024-06-26T07:10:39.2089353Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:10:39.2093173Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:10:39.2097033Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:10:39.2097549Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:10:39.2097911Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:10:39.2098294Z TEST_CONFIG=dynamic_inductor_torchbench 2024-06-26T07:10:39.2098731Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:10:39.2099209Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:10:39.2099677Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:10:39.2100126Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:10:39.2100547Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:10:39.2100997Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:39.2101510Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:10:39.2101985Z GITHUB_REF_TYPE=tag 2024-06-26T07:10:39.2102314Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:10:39.2102696Z NCCL_VERSION=2.17.1-1 2024-06-26T07:10:39.2103085Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.2103517Z XLA_CUDA= 2024-06-26T07:10:39.2104401Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:10:39.2104984Z *** 2024-06-26T07:10:39.2105366Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:10:39.2105909Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:10:39.2106389Z GITHUB_ACTIONS=true 2024-06-26T07:10:39.2106833Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:39.2107425Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:10:39.2107993Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:10:39.2108496Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.2109032Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:10:39.2109550Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.2110414Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:10:39.2111208Z UCC_HOME=/usr 2024-06-26T07:10:39.2111554Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:10:39.2111972Z VERBOSE_TEST_LOGS=False 2024-06-26T07:10:39.2112321Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:10:39.2112805Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:10:39.2113336Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:10:39.2113832Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:10:39.2114219Z SHARD_NUMBER=1 2024-06-26T07:10:39.2114537Z GITHUB_REF_PROTECTED=false 2024-06-26T07:10:39.2114911Z HOME=/var/lib/jenkins 2024-06-26T07:10:39.2115276Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:10:39.2115720Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:10:39.2116343Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:10:39.2116914Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:10:39.2117285Z CUDA_VERSION=12.1.1 2024-06-26T07:10:39.2117711Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:10:39.2118155Z NUM_TEST_SHARDS=2 2024-06-26T07:10:39.2118468Z UCX_HOME=/usr 2024-06-26T07:10:39.2118979Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:10:39.2120116Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.2121386Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:39.2122643Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.2123763Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:10:39.2124435Z GITHUB_EVENT_NAME=push 2024-06-26T07:10:39.2124774Z DASHBOARD_TAG= 2024-06-26T07:10:39.2125091Z GITHUB_RUN_ID=9673646808 2024-06-26T07:10:39.2125658Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:10:39.2126200Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:10:39.2127199Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.2128111Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:39.2128517Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:10:39.2128899Z PR_NUMBER= 2024-06-26T07:10:39.2129203Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:10:39.2129519Z VALGRIND=ON 2024-06-26T07:10:39.2129822Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:10:39.2130275Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:10:39.2130743Z TERM=xterm 2024-06-26T07:10:39.2131074Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:10:39.2131496Z INSTALLED_VISION=yes 2024-06-26T07:10:39.2131813Z BRANCH= 2024-06-26T07:10:39.2132111Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:10:39.2132518Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:10:39.2132953Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:10:39.2133709Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:10:39.2134466Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:10:39.2134958Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:10:39.2135440Z REENABLED_ISSUES= 2024-06-26T07:10:39.2135746Z SHLVL=1 2024-06-26T07:10:39.2136009Z MAX_JOBS=14 2024-06-26T07:10:39.2136321Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:10:39.2136685Z NVARCH=x86_64 2024-06-26T07:10:39.2136988Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:10:39.2137440Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.2137978Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:10:39.2138474Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:10:39.2139130Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:39.2139711Z GITHUB_JOB=test 2024-06-26T07:10:39.2140121Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:10:39.2140686Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:10:39.2141199Z NO_TEST_TIMEOUT=False 2024-06-26T07:10:39.2141533Z TD_DISTRIBUTED=False 2024-06-26T07:10:39.2141914Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:10:39.2142356Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:10:39.2142772Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:10:39.2143143Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:10:39.2143495Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:10:39.2157672Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:10:39.2158776Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:39.2159856Z GITHUB_BASE_REF= 2024-06-26T07:10:39.2160193Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:10:39.2160727Z CI=true 2024-06-26T07:10:39.2161067Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:39.2161591Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:10:39.2161978Z JOB_ID=26688911970 2024-06-26T07:10:39.2162295Z INSTALLED_PROTOBUF=yes 2024-06-26T07:10:39.2162625Z GITHUB_HEAD_REF= 2024-06-26T07:10:39.2162933Z GITHUB_ACTION_REF= 2024-06-26T07:10:39.2163378Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:10:39.2163849Z GITHUB_WORKFLOW=inductor 2024-06-26T07:10:39.2164209Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:10:39.2165119Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ad947ec7-cc94-4bf9-861f-49aef6bcbbc1 2024-06-26T07:10:39.2166222Z NO_TD=False 2024-06-26T07:10:39.2166532Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:10:39.2166898Z _=/usr/bin/env 2024-06-26T07:10:39.2167242Z + echo 'Testing pytorch' 2024-06-26T07:10:39.2167590Z Testing pytorch 2024-06-26T07:10:39.2167916Z + export LANG=C.UTF-8 2024-06-26T07:10:39.2168252Z + LANG=C.UTF-8 2024-06-26T07:10:39.2168581Z + PR_NUMBER= 2024-06-26T07:10:39.2168965Z + [[ dynamic_inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:10:39.2169545Z + [[ dynamic_inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:10:39.2170107Z + [[ dynamic_inductor_torchbench == \s\l\o\w ]] 2024-06-26T07:10:39.2170766Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:10:39.2171457Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:10:39.2172005Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:10:39.2172490Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:10:39.2172976Z + [[ dynamic_inductor_torchbench == *crossref* ]] 2024-06-26T07:10:39.2173575Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:10:39.2174213Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:10:39.2174861Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:10:39.2175424Z + pip_install --user ninja==1.10.2 2024-06-26T07:10:39.2175966Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:10:39.5763033Z Collecting ninja==1.10.2 2024-06-26T07:10:39.5950877Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:10:39.6115332Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:10:40.9112957Z Installing collected packages: ninja 2024-06-26T07:10:40.9180302Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:10:40.9181624Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:10:40.9251049Z Successfully installed ninja-1.10.2 2024-06-26T07:10:41.0190713Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:41.0192843Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:41.0194270Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:10:41.0194778Z + install_tlparse 2024-06-26T07:10:41.0195162Z + pip_install --user tlparse==0.3.7 2024-06-26T07:10:41.0195711Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:10:41.3742194Z Collecting tlparse==0.3.7 2024-06-26T07:10:41.3908283Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:10:41.4026423Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:10:42.7203282Z Installing collected packages: tlparse 2024-06-26T07:10:42.7568512Z Successfully installed tlparse-0.3.7 2024-06-26T07:10:42.8467042Z ++ python -m site --user-base 2024-06-26T07:10:42.8619849Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:42.8621837Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:10:42.8622501Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:10:42.8623174Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:10:42.8624451Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:10:42.8625602Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:10:42.8626300Z + cd test 2024-06-26T07:10:42.8626826Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:10:44.3466220Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:10:44.3467175Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:10:44.3468096Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:10:44.3468649Z + pushd test 2024-06-26T07:10:44.3468970Z ~/workspace/test ~/workspace 2024-06-26T07:10:44.3469495Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:10:45.8139012Z + CUDA_VERSION=12.1 2024-06-26T07:10:45.8139554Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:10:45.8139897Z + ISCUDA124= 2024-06-26T07:10:45.8140199Z + popd 2024-06-26T07:10:45.8140464Z ~/workspace 2024-06-26T07:10:45.8143327Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:10:45.8144142Z + [[ dynamic_inductor_torchbench == *dynamo_eager* ]] 2024-06-26T07:10:45.8144874Z + [[ dynamic_inductor_torchbench == *aot_eager* ]] 2024-06-26T07:10:45.8145459Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-26T07:10:45.8145990Z + [[ dynamic_inductor_torchbench == *inductor* ]] 2024-06-26T07:10:45.8146498Z + [[ dynamic_inductor_torchbench != *perf* ]] 2024-06-26T07:10:45.8147045Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:10:45.8147530Z + [[ dynamic_inductor_torchbench == *dynamic* ]] 2024-06-26T07:10:45.8148183Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-26T07:10:45.8148814Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-26T07:10:45.8149351Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:10:45.8171507Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:10:45.8172362Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:10:45.8173236Z + cd test 2024-06-26T07:10:45.8173689Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:10:47.1676754Z PyTorch built with: 2024-06-26T07:10:47.1677480Z - GCC 9.4 2024-06-26T07:10:47.1677872Z - C++ Version: 201703 2024-06-26T07:10:47.1678753Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:10:47.1679858Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:10:47.1680533Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:10:47.1681046Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:10:47.1681533Z - NNPACK is enabled 2024-06-26T07:10:47.1681919Z - CPU capability usage: AVX2 2024-06-26T07:10:47.1682331Z - CUDA Runtime 12.1 2024-06-26T07:10:47.1682846Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:10:47.1683453Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:10:47.1683904Z - Magma 2.6.1 2024-06-26T07:10:47.1691277Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:10:47.1698016Z 2024-06-26T07:10:47.4413391Z + cd test 2024-06-26T07:10:47.4414116Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:10:48.6495804Z ATen/Parallel: 2024-06-26T07:10:48.6496352Z at::get_num_threads() : 8 2024-06-26T07:10:48.6496787Z at::get_num_interop_threads() : 16 2024-06-26T07:10:48.6497262Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:10:48.6497699Z omp_get_max_threads() : 8 2024-06-26T07:10:48.6498764Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:10:48.6499635Z mkl_get_max_threads() : 8 2024-06-26T07:10:48.6500293Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:10:48.6500959Z std::thread::hardware_concurrency() : 16 2024-06-26T07:10:48.6501459Z Environment variables: 2024-06-26T07:10:48.6501805Z OMP_NUM_THREADS : [not set] 2024-06-26T07:10:48.6502245Z MKL_NUM_THREADS : [not set] 2024-06-26T07:10:48.6502625Z ATen parallel backend: OpenMP 2024-06-26T07:10:48.6502937Z 2024-06-26T07:10:48.8939571Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:10:48.8940317Z + [[ dynamic_inductor_torchbench == *backward* ]] 2024-06-26T07:10:48.8940832Z + [[ dynamic_inductor_torchbench == *xla* ]] 2024-06-26T07:10:48.8941415Z + [[ dynamic_inductor_torchbench == *executorch* ]] 2024-06-26T07:10:48.8942076Z + [[ dynamic_inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:10:48.8942764Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:10:48.8943332Z + [[ dynamic_inductor_torchbench == distributed ]] 2024-06-26T07:10:48.8944138Z + [[ dynamic_inductor_torchbench == *inductor_distributed* ]] 2024-06-26T07:10:48.8944806Z + [[ dynamic_inductor_torchbench == *inductor-halide* ]] 2024-06-26T07:10:48.8945489Z + [[ dynamic_inductor_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:10:48.8946091Z + [[ dynamic_inductor_torchbench == *huggingface* ]] 2024-06-26T07:10:48.8946610Z + [[ dynamic_inductor_torchbench == *timm* ]] 2024-06-26T07:10:48.8947115Z + [[ dynamic_inductor_torchbench == *torchbench* ]] 2024-06-26T07:10:48.8947659Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-26T07:10:48.8948145Z + install_torchaudio cuda 2024-06-26T07:10:48.8948493Z + local commit 2024-06-26T07:10:48.8948812Z ++ get_pinned_commit audio 2024-06-26T07:10:48.8949189Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:10:48.8953285Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:48.8953747Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:10:48.8954161Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:10:48.8955056Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:48.8956479Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:49.2080105Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:49.2084322Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-326bwcfl 2024-06-26T07:10:49.2102304Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-326bwcfl 2024-06-26T07:10:49.9190238Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:10:49.9204613Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:50.2959768Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:50.4187960Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:50.4188984Z Running command git submodule update --init --recursive -q 2024-06-26T07:10:52.5941407Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:10:52.5973726Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:10:52.6034277Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:10:52.6039799Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:10:52.6042046Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:10:52.6045342Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:10:52.6048024Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:10:52.6051016Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:10:52.6473202Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:10:52.6623162Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:10:52.6692379Z Building wheels for collected packages: torchaudio 2024-06-26T07:13:19.0545699Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-06-26T07:13:19.0575840Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308806 sha256=0b59c3998d7dd046d09ea7474b24047b9be41479ae43c6767cb1f0176ac0d6d3 2024-06-26T07:13:19.0578636Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:13:19.0603759Z Successfully built torchaudio 2024-06-26T07:13:20.2688703Z Installing collected packages: torchaudio 2024-06-26T07:13:20.4675132Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:13:20.6022577Z + install_torchtext 2024-06-26T07:13:20.6023044Z + local data_commit 2024-06-26T07:13:20.6025643Z + local text_commit 2024-06-26T07:13:20.6026741Z ++ get_pinned_commit data 2024-06-26T07:13:20.6027193Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:13:20.6037297Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:20.6039034Z ++ get_pinned_commit text 2024-06-26T07:13:20.6039659Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:13:20.6047799Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:20.6049145Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:20.6050582Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:20.9169631Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:20.9173512Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-k3e7t2lu 2024-06-26T07:13:20.9188005Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-k3e7t2lu 2024-06-26T07:13:21.3448464Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:13:21.3467232Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:21.5692238Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:21.7046414Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:21.7047457Z Running command git submodule update --init --recursive -q 2024-06-26T07:14:34.1109574Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:14:34.1159593Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:14:34.1162669Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:14:34.1168077Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:14:34.1228514Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:14:34.1233774Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:14:34.1236796Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:14:34.1239794Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:14:34.1242825Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:14:34.1245856Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:14:34.1432633Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:14:34.1437726Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:14:34.1443850Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:14:34.1902981Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:14:34.2062373Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:14:34.2141260Z Building wheels for collected packages: torchdata 2024-06-26T07:14:35.9678639Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:14:35.9685714Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=7195310c0c01efb9a0d9b199fa0d145becd8e61472434d1dceb1b45d0f56275c 2024-06-26T07:14:35.9687402Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:14:35.9713804Z Successfully built torchdata 2024-06-26T07:14:37.1564532Z Installing collected packages: torchdata 2024-06-26T07:14:37.2452892Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:14:39.7506371Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:14:39.7507933Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:14:40.0636539Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:14:40.0641038Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-c8773g6h 2024-06-26T07:14:40.0660071Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-c8773g6h 2024-06-26T07:14:40.8847204Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:14:40.8866005Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:14:41.3385249Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:14:41.4877906Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:14:41.4879362Z Running command git submodule update --init --recursive -q 2024-06-26T07:14:48.5946318Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:14:48.5996339Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:14:48.5999402Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:14:48.6002185Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:14:48.6005010Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:14:48.6008382Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:14:48.6067628Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:14:48.6072603Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:14:48.6078893Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:14:48.6083683Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:14:48.6139147Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:14:48.6145078Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:14:48.6148430Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:14:48.6150530Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:14:48.6153353Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:14:48.6155973Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:14:48.6828030Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:14:48.6994303Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:14:48.7075092Z Building wheels for collected packages: torchtext 2024-06-26T07:15:25.1877409Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:15:25.1905296Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045304 sha256=0f8903b6db5a7d7862a4116e0d205d3f48d2cbcb0af5c47ca2efd46c9decf27e 2024-06-26T07:15:25.1907129Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:15:25.1933220Z Successfully built torchtext 2024-06-26T07:15:26.3683840Z Installing collected packages: torchtext 2024-06-26T07:15:26.4720427Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:15:26.5958822Z + install_torchvision 2024-06-26T07:15:26.5959187Z + local orig_preload 2024-06-26T07:15:26.5961890Z + local commit 2024-06-26T07:15:26.5962488Z ++ get_pinned_commit vision 2024-06-26T07:15:26.5962924Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:15:26.5971561Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:26.5972028Z + orig_preload= 2024-06-26T07:15:26.5972389Z + '[' -n '' ']' 2024-06-26T07:15:26.5973230Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:26.5974663Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:26.9108501Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:26.9113111Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-qsi0fgd9 2024-06-26T07:15:26.9130460Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-qsi0fgd9 2024-06-26T07:15:28.3025367Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:15:28.3040794Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:29.5252529Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:29.7553574Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:32.0711329Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:15:32.0759494Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:15:32.0762880Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:15:32.0768753Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:15:32.0976953Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:15:32.0983743Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:15:32.0990490Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:15:32.0992303Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:15:32.0994563Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:15:32.0996903Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:15:32.1430895Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:15:32.1596799Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:15:32.1671495Z Building wheels for collected packages: torchvision 2024-06-26T07:16:45.9756126Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:16:45.9782927Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112797 sha256=7281b13c7049384b76dd9813e29b9540cc63b1603518c15dc8af199820e9c0fa 2024-06-26T07:16:45.9785560Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:16:45.9820674Z Successfully built torchvision 2024-06-26T07:16:47.1714730Z Installing collected packages: torchvision 2024-06-26T07:16:47.5638246Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:16:47.6743718Z + '[' -n '' ']' 2024-06-26T07:16:47.6744415Z + id=0 2024-06-26T07:16:47.6744791Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:16:47.6745375Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:16:48.1029669Z Collecting opencv-python==4.8.0.74 2024-06-26T07:16:48.1182238Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:16:48.1368921Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:16:48.1429947Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:16:50.0561933Z Installing collected packages: opencv-python 2024-06-26T07:16:50.8747406Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:16:50.9729875Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:16:50.9731296Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:16:50.9732114Z + [[ dynamic_inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:16:50.9732763Z + checkout_install_torchbench 2024-06-26T07:16:50.9733133Z + local commit 2024-06-26T07:16:50.9733464Z ++ get_pinned_commit torchbench 2024-06-26T07:16:50.9733893Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:16:50.9744565Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:16:50.9745359Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:16:50.9753547Z Cloning into 'torchbench'... 2024-06-26T07:16:51.1110555Z remote: Enumerating objects: 30187, done. 2024-06-26T07:16:51.1111661Z remote: Counting objects: 0% (1/546) 2024-06-26T07:16:51.1112352Z remote: Counting objects: 1% (6/546) 2024-06-26T07:16:51.1113034Z remote: Counting objects: 2% (11/546) 2024-06-26T07:16:51.1113649Z remote: Counting objects: 3% (17/546) 2024-06-26T07:16:51.1114266Z remote: Counting objects: 4% (22/546) 2024-06-26T07:16:51.1116133Z remote: Counting objects: 5% (28/546) 2024-06-26T07:16:51.1116931Z remote: Counting objects: 6% (33/546) 2024-06-26T07:16:51.1117543Z remote: Counting objects: 7% (39/546) 2024-06-26T07:16:51.1118050Z remote: Counting objects: 8% (44/546) 2024-06-26T07:16:51.1118550Z remote: Counting objects: 9% (50/546) 2024-06-26T07:16:51.1119139Z remote: Counting objects: 10% (55/546) 2024-06-26T07:16:51.1119870Z remote: Counting objects: 11% (61/546) 2024-06-26T07:16:51.1120541Z remote: Counting objects: 12% (66/546) 2024-06-26T07:16:51.1121092Z remote: Counting objects: 13% (71/546) 2024-06-26T07:16:51.1121656Z remote: Counting objects: 14% (77/546) 2024-06-26T07:16:51.1122168Z remote: Counting objects: 15% (82/546) 2024-06-26T07:16:51.1122710Z remote: Counting objects: 16% (88/546) 2024-06-26T07:16:51.1123266Z remote: Counting objects: 17% (93/546) 2024-06-26T07:16:51.1123763Z remote: Counting objects: 18% (99/546) 2024-06-26T07:16:51.1124325Z remote: Counting objects: 19% (104/546) 2024-06-26T07:16:51.1124897Z remote: Counting objects: 20% (110/546) 2024-06-26T07:16:51.1125408Z remote: Counting objects: 21% (115/546) 2024-06-26T07:16:51.1126098Z remote: Counting objects: 22% (121/546) 2024-06-26T07:16:51.1126672Z remote: Counting objects: 23% (126/546) 2024-06-26T07:16:51.1127210Z remote: Counting objects: 24% (132/546) 2024-06-26T07:16:51.1127754Z remote: Counting objects: 25% (137/546) 2024-06-26T07:16:51.1128325Z remote: Counting objects: 26% (142/546) 2024-06-26T07:16:51.1128837Z remote: Counting objects: 27% (148/546) 2024-06-26T07:16:51.1129398Z remote: Counting objects: 28% (153/546) 2024-06-26T07:16:51.1129966Z remote: Counting objects: 29% (159/546) 2024-06-26T07:16:51.1130508Z remote: Counting objects: 30% (164/546) 2024-06-26T07:16:51.1131087Z remote: Counting objects: 31% (170/546) 2024-06-26T07:16:51.1131654Z remote: Counting objects: 32% (175/546) 2024-06-26T07:16:51.1132172Z remote: Counting objects: 33% (181/546) 2024-06-26T07:16:51.1132721Z remote: Counting objects: 34% (186/546) 2024-06-26T07:16:51.1133281Z remote: Counting objects: 35% (192/546) 2024-06-26T07:16:51.1133782Z remote: Counting objects: 36% (197/546) 2024-06-26T07:16:51.1134342Z remote: Counting objects: 37% (203/546) 2024-06-26T07:16:51.1134902Z remote: Counting objects: 38% (208/546) 2024-06-26T07:16:51.1135404Z remote: Counting objects: 39% (213/546) 2024-06-26T07:16:51.1135968Z remote: Counting objects: 40% (219/546) 2024-06-26T07:16:51.1136510Z remote: Counting objects: 41% (224/546) 2024-06-26T07:16:51.1137016Z remote: Counting objects: 42% (230/546) 2024-06-26T07:16:51.1137569Z remote: Counting objects: 43% (235/546) 2024-06-26T07:16:51.1138107Z remote: Counting objects: 44% (241/546) 2024-06-26T07:16:51.1138625Z remote: Counting objects: 45% (246/546) 2024-06-26T07:16:51.1139180Z remote: Counting objects: 46% (252/546) 2024-06-26T07:16:51.1139717Z remote: Counting objects: 47% (257/546) 2024-06-26T07:16:51.1140248Z remote: Counting objects: 48% (263/546) 2024-06-26T07:16:51.1140807Z remote: Counting objects: 49% (268/546) 2024-06-26T07:16:51.1141334Z remote: Counting objects: 50% (273/546) 2024-06-26T07:16:51.1141880Z remote: Counting objects: 51% (279/546) 2024-06-26T07:16:51.1142723Z remote: Counting objects: 52% (284/546) 2024-06-26T07:16:51.1143301Z remote: Counting objects: 53% (290/546) 2024-06-26T07:16:51.1144279Z remote: Counting objects: 54% (295/546) 2024-06-26T07:16:51.1144865Z remote: Counting objects: 55% (301/546) 2024-06-26T07:16:51.1145456Z remote: Counting objects: 56% (306/546) 2024-06-26T07:16:51.1146032Z remote: Counting objects: 57% (312/546) 2024-06-26T07:16:51.1146558Z remote: Counting objects: 58% (317/546) 2024-06-26T07:16:51.1147120Z remote: Counting objects: 59% (323/546) 2024-06-26T07:16:51.1147683Z remote: Counting objects: 60% (328/546) 2024-06-26T07:16:51.1148213Z remote: Counting objects: 61% (334/546) 2024-06-26T07:16:51.1148763Z remote: Counting objects: 62% (339/546) 2024-06-26T07:16:51.1149333Z remote: Counting objects: 63% (344/546) 2024-06-26T07:16:51.1149851Z remote: Counting objects: 64% (350/546) 2024-06-26T07:16:51.1150406Z remote: Counting objects: 65% (355/546) 2024-06-26T07:16:51.1150956Z remote: Counting objects: 66% (361/546) 2024-06-26T07:16:51.1151496Z remote: Counting objects: 67% (366/546) 2024-06-26T07:16:51.1152009Z remote: Counting objects: 68% (372/546) 2024-06-26T07:16:51.1152578Z remote: Counting objects: 69% (377/546) 2024-06-26T07:16:51.1153097Z remote: Counting objects: 70% (383/546) 2024-06-26T07:16:51.1153606Z remote: Counting objects: 71% (388/546) 2024-06-26T07:16:51.1154113Z remote: Counting objects: 72% (394/546) 2024-06-26T07:16:51.1154668Z remote: Counting objects: 73% (399/546) 2024-06-26T07:16:51.1155186Z remote: Counting objects: 74% (405/546) 2024-06-26T07:16:51.1155701Z remote: Counting objects: 75% (410/546) 2024-06-26T07:16:51.1156202Z remote: Counting objects: 76% (415/546) 2024-06-26T07:16:51.1156717Z remote: Counting objects: 77% (421/546) 2024-06-26T07:16:51.1157231Z remote: Counting objects: 78% (426/546) 2024-06-26T07:16:51.1157738Z remote: Counting objects: 79% (432/546) 2024-06-26T07:16:51.1158259Z remote: Counting objects: 80% (437/546) 2024-06-26T07:16:51.1158833Z remote: Counting objects: 81% (443/546) 2024-06-26T07:16:51.1159350Z remote: Counting objects: 82% (448/546) 2024-06-26T07:16:51.1159854Z remote: Counting objects: 83% (454/546) 2024-06-26T07:16:51.1160363Z remote: Counting objects: 84% (459/546) 2024-06-26T07:16:51.1160904Z remote: Counting objects: 85% (465/546) 2024-06-26T07:16:51.1161446Z remote: Counting objects: 86% (470/546) 2024-06-26T07:16:51.1161954Z remote: Counting objects: 87% (476/546) 2024-06-26T07:16:51.1162462Z remote: Counting objects: 88% (481/546) 2024-06-26T07:16:51.1162989Z remote: Counting objects: 89% (486/546) 2024-06-26T07:16:51.1163504Z remote: Counting objects: 90% (492/546) 2024-06-26T07:16:51.1164007Z remote: Counting objects: 91% (497/546) 2024-06-26T07:16:51.1164523Z remote: Counting objects: 92% (503/546) 2024-06-26T07:16:51.1165052Z remote: Counting objects: 93% (508/546) 2024-06-26T07:16:51.1165638Z remote: Counting objects: 94% (514/546) 2024-06-26T07:16:51.1166227Z remote: Counting objects: 95% (519/546) 2024-06-26T07:16:51.1166763Z remote: Counting objects: 96% (525/546) 2024-06-26T07:16:51.1167282Z remote: Counting objects: 97% (530/546) 2024-06-26T07:16:51.1167796Z remote: Counting objects: 98% (536/546) 2024-06-26T07:16:51.1168312Z remote: Counting objects: 99% (541/546) 2024-06-26T07:16:51.1168826Z remote: Counting objects: 100% (546/546) 2024-06-26T07:16:51.1169379Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:16:51.1169942Z remote: Compressing objects: 0% (1/318) 2024-06-26T07:16:51.1170543Z remote: Compressing objects: 1% (4/318) 2024-06-26T07:16:51.1171153Z remote: Compressing objects: 2% (7/318) 2024-06-26T07:16:51.1171693Z remote: Compressing objects: 3% (10/318) 2024-06-26T07:16:51.1172227Z remote: Compressing objects: 4% (13/318) 2024-06-26T07:16:51.1172764Z remote: Compressing objects: 5% (16/318) 2024-06-26T07:16:51.1173410Z remote: Compressing objects: 6% (20/318) 2024-06-26T07:16:51.1174015Z remote: Compressing objects: 7% (23/318) 2024-06-26T07:16:51.1174544Z remote: Compressing objects: 8% (26/318) 2024-06-26T07:16:51.1175078Z remote: Compressing objects: 9% (29/318) 2024-06-26T07:16:51.1175612Z remote: Compressing objects: 10% (32/318) 2024-06-26T07:16:51.1176157Z remote: Compressing objects: 11% (35/318) 2024-06-26T07:16:51.1176684Z remote: Compressing objects: 12% (39/318) 2024-06-26T07:16:51.1177723Z remote: Compressing objects: 13% (42/318) 2024-06-26T07:16:51.1179552Z remote: Compressing objects: 14% (45/318) 2024-06-26T07:16:51.1180220Z remote: Compressing objects: 15% (48/318) 2024-06-26T07:16:51.1180911Z remote: Compressing objects: 16% (51/318) 2024-06-26T07:16:51.1181527Z remote: Compressing objects: 17% (55/318) 2024-06-26T07:16:51.1182086Z remote: Compressing objects: 18% (58/318) 2024-06-26T07:16:51.1182789Z remote: Compressing objects: 19% (61/318) 2024-06-26T07:16:51.1183345Z remote: Compressing objects: 20% (64/318) 2024-06-26T07:16:51.1184169Z remote: Compressing objects: 21% (67/318) 2024-06-26T07:16:51.1184833Z remote: Compressing objects: 22% (70/318) 2024-06-26T07:16:51.1185416Z remote: Compressing objects: 23% (74/318) 2024-06-26T07:16:51.1185954Z remote: Compressing objects: 24% (77/318) 2024-06-26T07:16:51.1186476Z remote: Compressing objects: 25% (80/318) 2024-06-26T07:16:51.1187045Z remote: Compressing objects: 26% (83/318) 2024-06-26T07:16:51.1187761Z remote: Compressing objects: 27% (86/318) 2024-06-26T07:16:51.1188434Z remote: Compressing objects: 28% (90/318) 2024-06-26T07:16:51.1188966Z remote: Compressing objects: 29% (93/318) 2024-06-26T07:16:51.1189484Z remote: Compressing objects: 30% (96/318) 2024-06-26T07:16:51.1190008Z remote: Compressing objects: 31% (99/318) 2024-06-26T07:16:51.1190567Z remote: Compressing objects: 32% (102/318) 2024-06-26T07:16:51.1191121Z remote: Compressing objects: 33% (105/318) 2024-06-26T07:16:51.1191787Z remote: Compressing objects: 34% (109/318) 2024-06-26T07:16:51.1192428Z remote: Compressing objects: 35% (112/318) 2024-06-26T07:16:51.1192978Z remote: Compressing objects: 36% (115/318) 2024-06-26T07:16:51.1193704Z remote: Compressing objects: 37% (118/318) 2024-06-26T07:16:51.1194442Z remote: Compressing objects: 38% (121/318) 2024-06-26T07:16:51.1195096Z remote: Compressing objects: 39% (125/318) 2024-06-26T07:16:51.1195622Z remote: Compressing objects: 40% (128/318) 2024-06-26T07:16:51.1196158Z remote: Compressing objects: 41% (131/318) 2024-06-26T07:16:51.1196691Z remote: Compressing objects: 42% (134/318) 2024-06-26T07:16:51.1197219Z remote: Compressing objects: 43% (137/318) 2024-06-26T07:16:51.1197851Z remote: Compressing objects: 44% (140/318) 2024-06-26T07:16:51.1198386Z remote: Compressing objects: 45% (144/318) 2024-06-26T07:16:51.1198918Z remote: Compressing objects: 46% (147/318) 2024-06-26T07:16:51.1199457Z remote: Compressing objects: 47% (150/318) 2024-06-26T07:16:51.1199983Z remote: Compressing objects: 48% (153/318) 2024-06-26T07:16:51.1200544Z remote: Compressing objects: 49% (156/318) 2024-06-26T07:16:51.1201096Z remote: Compressing objects: 50% (159/318) 2024-06-26T07:16:51.1201623Z remote: Compressing objects: 51% (163/318) 2024-06-26T07:16:51.1202142Z remote: Compressing objects: 52% (166/318) 2024-06-26T07:16:51.1202671Z remote: Compressing objects: 53% (169/318) 2024-06-26T07:16:51.1203201Z remote: Compressing objects: 54% (172/318) 2024-06-26T07:16:51.1203722Z remote: Compressing objects: 55% (175/318) 2024-06-26T07:16:51.1204247Z remote: Compressing objects: 56% (179/318) 2024-06-26T07:16:51.1204773Z remote: Compressing objects: 57% (182/318) 2024-06-26T07:16:51.1205368Z remote: Compressing objects: 58% (185/318) 2024-06-26T07:16:51.1206013Z remote: Compressing objects: 59% (188/318) 2024-06-26T07:16:51.1206678Z remote: Compressing objects: 60% (191/318) 2024-06-26T07:16:51.1207212Z remote: Compressing objects: 61% (194/318) 2024-06-26T07:16:51.1207820Z remote: Compressing objects: 62% (198/318) 2024-06-26T07:16:51.1208341Z remote: Compressing objects: 63% (201/318) 2024-06-26T07:16:51.1208875Z remote: Compressing objects: 64% (204/318) 2024-06-26T07:16:51.1209426Z remote: Compressing objects: 65% (207/318) 2024-06-26T07:16:51.1209954Z remote: Compressing objects: 66% (210/318) 2024-06-26T07:16:51.1210474Z remote: Compressing objects: 67% (214/318) 2024-06-26T07:16:51.1211001Z remote: Compressing objects: 68% (217/318) 2024-06-26T07:16:51.1211528Z remote: Compressing objects: 69% (220/318) 2024-06-26T07:16:51.1212053Z remote: Compressing objects: 70% (223/318) 2024-06-26T07:16:51.1212575Z remote: Compressing objects: 71% (226/318) 2024-06-26T07:16:51.1213104Z remote: Compressing objects: 72% (229/318) 2024-06-26T07:16:51.1213635Z remote: Compressing objects: 73% (233/318) 2024-06-26T07:16:51.1214170Z remote: Compressing objects: 74% (236/318) 2024-06-26T07:16:51.1214700Z remote: Compressing objects: 75% (239/318) 2024-06-26T07:16:51.1215226Z remote: Compressing objects: 76% (242/318) 2024-06-26T07:16:51.1215752Z remote: Compressing objects: 77% (245/318) 2024-06-26T07:16:51.1216282Z remote: Compressing objects: 78% (249/318) 2024-06-26T07:16:51.1216801Z remote: Compressing objects: 79% (252/318) 2024-06-26T07:16:51.1217342Z remote: Compressing objects: 80% (255/318) 2024-06-26T07:16:51.1217873Z remote: Compressing objects: 81% (258/318) 2024-06-26T07:16:51.1218398Z remote: Compressing objects: 82% (261/318) 2024-06-26T07:16:51.1218920Z remote: Compressing objects: 83% (264/318) 2024-06-26T07:16:51.1219454Z remote: Compressing objects: 84% (268/318) 2024-06-26T07:16:51.1219986Z remote: Compressing objects: 85% (271/318) 2024-06-26T07:16:51.1220514Z remote: Compressing objects: 86% (274/318) 2024-06-26T07:16:51.1221054Z remote: Compressing objects: 87% (277/318) 2024-06-26T07:16:51.1221584Z remote: Compressing objects: 88% (280/318) 2024-06-26T07:16:51.1222116Z remote: Compressing objects: 89% (284/318) 2024-06-26T07:16:51.1222645Z remote: Compressing objects: 90% (287/318) 2024-06-26T07:16:51.1223174Z remote: Compressing objects: 91% (290/318) 2024-06-26T07:16:51.1223723Z remote: Compressing objects: 92% (293/318) 2024-06-26T07:16:51.1224589Z remote: Compressing objects: 93% (296/318) 2024-06-26T07:16:51.1225113Z remote: Compressing objects: 94% (299/318) 2024-06-26T07:16:51.1225645Z remote: Compressing objects: 95% (303/318) 2024-06-26T07:16:51.1226172Z remote: Compressing objects: 96% (306/318) 2024-06-26T07:16:51.1226705Z remote: Compressing objects: 97% (309/318) 2024-06-26T07:16:51.1227227Z remote: Compressing objects: 98% (312/318) 2024-06-26T07:16:51.1227758Z remote: Compressing objects: 99% (315/318) 2024-06-26T07:16:51.1228283Z remote: Compressing objects: 100% (318/318) 2024-06-26T07:16:51.1228849Z remote: Compressing objects: 100% (318/318), done. 2024-06-26T07:16:51.1310004Z Receiving objects: 0% (1/30187) 2024-06-26T07:16:51.1383541Z Receiving objects: 1% (302/30187) 2024-06-26T07:16:51.1450300Z Receiving objects: 2% (604/30187) 2024-06-26T07:16:51.1494853Z Receiving objects: 3% (906/30187) 2024-06-26T07:16:51.1527107Z Receiving objects: 4% (1208/30187) 2024-06-26T07:16:51.1567230Z Receiving objects: 5% (1510/30187) 2024-06-26T07:16:51.1600782Z Receiving objects: 6% (1812/30187) 2024-06-26T07:16:51.1639739Z Receiving objects: 7% (2114/30187) 2024-06-26T07:16:51.1671828Z Receiving objects: 8% (2415/30187) 2024-06-26T07:16:51.1700660Z Receiving objects: 9% (2717/30187) 2024-06-26T07:16:51.1734606Z Receiving objects: 10% (3019/30187) 2024-06-26T07:16:51.1794339Z Receiving objects: 11% (3321/30187) 2024-06-26T07:16:51.1857656Z Receiving objects: 12% (3623/30187) 2024-06-26T07:16:51.1913713Z Receiving objects: 13% (3925/30187) 2024-06-26T07:16:51.2083615Z Receiving objects: 14% (4227/30187) 2024-06-26T07:16:51.2184455Z Receiving objects: 15% (4529/30187) 2024-06-26T07:16:51.2270699Z Receiving objects: 16% (4830/30187) 2024-06-26T07:16:51.2323835Z Receiving objects: 17% (5132/30187) 2024-06-26T07:16:51.2360633Z Receiving objects: 18% (5434/30187) 2024-06-26T07:16:51.2388683Z Receiving objects: 19% (5736/30187) 2024-06-26T07:16:51.2453322Z Receiving objects: 20% (6038/30187) 2024-06-26T07:16:51.2472087Z Receiving objects: 21% (6340/30187) 2024-06-26T07:16:51.2490298Z Receiving objects: 22% (6642/30187) 2024-06-26T07:16:51.3286313Z Receiving objects: 23% (6944/30187) 2024-06-26T07:16:52.1239195Z Receiving objects: 24% (7245/30187) 2024-06-26T07:16:52.2593660Z Receiving objects: 24% (7446/30187), 78.41 MiB | 78.48 MiB/s 2024-06-26T07:16:52.7520402Z Receiving objects: 25% (7547/30187), 78.41 MiB | 78.48 MiB/s 2024-06-26T07:16:52.8179174Z Receiving objects: 26% (7849/30187), 115.73 MiB | 77.20 MiB/s 2024-06-26T07:16:52.8837220Z Receiving objects: 27% (8151/30187), 115.73 MiB | 77.20 MiB/s 2024-06-26T07:16:52.9564638Z Receiving objects: 28% (8453/30187), 115.73 MiB | 77.20 MiB/s 2024-06-26T07:16:53.0367709Z Receiving objects: 29% (8755/30187), 115.73 MiB | 77.20 MiB/s 2024-06-26T07:16:53.1169271Z Receiving objects: 30% (9057/30187), 115.73 MiB | 77.20 MiB/s 2024-06-26T07:16:53.1237692Z Receiving objects: 31% (9358/30187), 115.73 MiB | 77.20 MiB/s 2024-06-26T07:16:53.1931307Z Receiving objects: 31% (9386/30187), 154.12 MiB | 77.10 MiB/s 2024-06-26T07:16:53.2629246Z Receiving objects: 32% (9660/30187), 154.12 MiB | 77.10 MiB/s 2024-06-26T07:16:53.4881427Z Receiving objects: 33% (9962/30187), 154.12 MiB | 77.10 MiB/s 2024-06-26T07:16:53.5604027Z Receiving objects: 34% (10264/30187), 154.12 MiB | 77.10 MiB/s 2024-06-26T07:16:53.5723311Z Receiving objects: 35% (10566/30187), 154.12 MiB | 77.10 MiB/s 2024-06-26T07:16:53.7676000Z Receiving objects: 36% (10868/30187), 154.12 MiB | 77.10 MiB/s 2024-06-26T07:16:54.0377285Z Receiving objects: 37% (11170/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0580470Z Receiving objects: 38% (11472/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0594133Z Receiving objects: 39% (11773/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0619131Z Receiving objects: 40% (12075/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0632149Z Receiving objects: 41% (12377/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0644801Z Receiving objects: 42% (12679/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0655435Z Receiving objects: 43% (12981/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0664507Z Receiving objects: 44% (13283/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0683892Z Receiving objects: 45% (13585/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0695337Z Receiving objects: 46% (13887/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0699081Z Receiving objects: 47% (14188/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0705080Z Receiving objects: 48% (14490/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0725868Z Receiving objects: 49% (14792/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0741949Z Receiving objects: 50% (15094/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0752186Z Receiving objects: 51% (15396/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0946435Z Receiving objects: 52% (15698/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0950685Z Receiving objects: 53% (16000/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0957716Z Receiving objects: 54% (16301/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0963452Z Receiving objects: 55% (16603/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0977639Z Receiving objects: 56% (16905/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0984831Z Receiving objects: 57% (17207/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.0994537Z Receiving objects: 58% (17509/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1003287Z Receiving objects: 59% (17811/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1016044Z Receiving objects: 60% (18113/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1024283Z Receiving objects: 61% (18415/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1033672Z Receiving objects: 62% (18716/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1160098Z Receiving objects: 63% (19018/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1166102Z Receiving objects: 64% (19320/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1172015Z Receiving objects: 65% (19622/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1177979Z Receiving objects: 66% (19924/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1186936Z Receiving objects: 67% (20226/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1194178Z Receiving objects: 68% (20528/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1237781Z Receiving objects: 69% (20830/30187), 185.70 MiB | 74.31 MiB/s 2024-06-26T07:16:54.1436856Z Receiving objects: 69% (20978/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1824001Z Receiving objects: 70% (21131/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1829773Z Receiving objects: 71% (21433/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1840656Z Receiving objects: 72% (21735/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1853209Z Receiving objects: 73% (22037/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1865873Z Receiving objects: 74% (22339/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1875638Z Receiving objects: 75% (22641/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1883906Z Receiving objects: 76% (22943/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1901916Z Receiving objects: 77% (23244/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1908108Z Receiving objects: 78% (23546/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1915540Z Receiving objects: 79% (23848/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.1937180Z Receiving objects: 80% (24150/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.3869485Z Receiving objects: 81% (24452/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4079172Z Receiving objects: 82% (24754/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4110974Z Receiving objects: 83% (25056/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4120735Z Receiving objects: 84% (25358/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4131674Z Receiving objects: 85% (25659/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4160502Z Receiving objects: 86% (25961/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4189141Z Receiving objects: 87% (26263/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4316577Z Receiving objects: 88% (26565/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4322375Z Receiving objects: 89% (26867/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4329122Z Receiving objects: 90% (27169/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4334811Z Receiving objects: 91% (27471/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4341043Z Receiving objects: 92% (27773/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4346771Z Receiving objects: 93% (28074/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4456308Z Receiving objects: 94% (28376/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4471212Z Receiving objects: 95% (28678/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4480985Z Receiving objects: 96% (28980/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4498612Z Receiving objects: 97% (29282/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4513278Z Receiving objects: 98% (29584/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4553028Z Receiving objects: 99% (29886/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4553938Z remote: Total 30187 (delta 256), reused 486 (delta 216), pack-reused 29641 2024-06-26T07:16:54.4570359Z Receiving objects: 100% (30187/30187), 225.50 MiB | 75.19 MiB/s 2024-06-26T07:16:54.4571210Z Receiving objects: 100% (30187/30187), 245.25 MiB | 73.58 MiB/s, done. 2024-06-26T07:16:54.4598412Z Resolving deltas: 0% (0/16205) 2024-06-26T07:16:54.4612584Z Resolving deltas: 1% (164/16205) 2024-06-26T07:16:54.4622998Z Resolving deltas: 2% (409/16205) 2024-06-26T07:16:54.4637970Z Resolving deltas: 3% (506/16205) 2024-06-26T07:16:54.4659189Z Resolving deltas: 4% (649/16205) 2024-06-26T07:16:54.4669780Z Resolving deltas: 5% (822/16205) 2024-06-26T07:16:54.4678380Z Resolving deltas: 6% (973/16205) 2024-06-26T07:16:54.4688926Z Resolving deltas: 7% (1141/16205) 2024-06-26T07:16:54.4698348Z Resolving deltas: 8% (1302/16205) 2024-06-26T07:16:54.4707974Z Resolving deltas: 9% (1477/16205) 2024-06-26T07:16:54.4718886Z Resolving deltas: 10% (1636/16205) 2024-06-26T07:16:54.4730275Z Resolving deltas: 11% (1785/16205) 2024-06-26T07:16:54.4739178Z Resolving deltas: 12% (1948/16205) 2024-06-26T07:16:54.4752267Z Resolving deltas: 13% (2109/16205) 2024-06-26T07:16:54.4761953Z Resolving deltas: 14% (2269/16205) 2024-06-26T07:16:54.4780366Z Resolving deltas: 15% (2442/16205) 2024-06-26T07:16:54.4796156Z Resolving deltas: 16% (2620/16205) 2024-06-26T07:16:54.4812356Z Resolving deltas: 17% (2757/16205) 2024-06-26T07:16:54.4814656Z Resolving deltas: 18% (3038/16205) 2024-06-26T07:16:54.4823025Z Resolving deltas: 19% (3084/16205) 2024-06-26T07:16:54.4843460Z Resolving deltas: 20% (3266/16205) 2024-06-26T07:16:54.4871083Z Resolving deltas: 21% (3417/16205) 2024-06-26T07:16:54.4894681Z Resolving deltas: 22% (3566/16205) 2024-06-26T07:16:54.4895340Z Resolving deltas: 23% (3866/16205) 2024-06-26T07:16:54.4905798Z Resolving deltas: 24% (3890/16205) 2024-06-26T07:16:54.4913925Z Resolving deltas: 25% (4069/16205) 2024-06-26T07:16:54.4924523Z Resolving deltas: 26% (4222/16205) 2024-06-26T07:16:54.4933209Z Resolving deltas: 27% (4408/16205) 2024-06-26T07:16:54.4974235Z Resolving deltas: 28% (4539/16205) 2024-06-26T07:16:54.4992583Z Resolving deltas: 29% (4722/16205) 2024-06-26T07:16:54.5005101Z Resolving deltas: 30% (4865/16205) 2024-06-26T07:16:54.5020442Z Resolving deltas: 31% (5025/16205) 2024-06-26T07:16:54.5035314Z Resolving deltas: 32% (5200/16205) 2024-06-26T07:16:54.5050496Z Resolving deltas: 33% (5348/16205) 2024-06-26T07:16:54.5060072Z Resolving deltas: 34% (5532/16205) 2024-06-26T07:16:54.5069321Z Resolving deltas: 35% (5706/16205) 2024-06-26T07:16:54.5080051Z Resolving deltas: 36% (5834/16205) 2024-06-26T07:16:54.5101365Z Resolving deltas: 37% (5999/16205) 2024-06-26T07:16:54.5115215Z Resolving deltas: 38% (6172/16205) 2024-06-26T07:16:54.5135701Z Resolving deltas: 39% (6324/16205) 2024-06-26T07:16:54.5142200Z Resolving deltas: 40% (6530/16205) 2024-06-26T07:16:54.5163817Z Resolving deltas: 41% (6662/16205) 2024-06-26T07:16:54.5175116Z Resolving deltas: 42% (6825/16205) 2024-06-26T07:16:54.5182435Z Resolving deltas: 43% (7026/16205) 2024-06-26T07:16:54.5203495Z Resolving deltas: 44% (7166/16205) 2024-06-26T07:16:54.5210092Z Resolving deltas: 45% (7350/16205) 2024-06-26T07:16:54.5218473Z Resolving deltas: 46% (7474/16205) 2024-06-26T07:16:54.5228713Z Resolving deltas: 47% (7627/16205) 2024-06-26T07:16:54.5249123Z Resolving deltas: 48% (7784/16205) 2024-06-26T07:16:54.5268333Z Resolving deltas: 49% (7942/16205) 2024-06-26T07:16:54.5280539Z Resolving deltas: 50% (8109/16205) 2024-06-26T07:16:54.5304383Z Resolving deltas: 51% (8268/16205) 2024-06-26T07:16:54.5313615Z Resolving deltas: 52% (8431/16205) 2024-06-26T07:16:54.5320379Z Resolving deltas: 53% (8589/16205) 2024-06-26T07:16:54.5331095Z Resolving deltas: 54% (8755/16205) 2024-06-26T07:16:54.5345515Z Resolving deltas: 55% (8919/16205) 2024-06-26T07:16:54.5355093Z Resolving deltas: 56% (9088/16205) 2024-06-26T07:16:54.5370222Z Resolving deltas: 57% (9238/16205) 2024-06-26T07:16:54.5378860Z Resolving deltas: 58% (9400/16205) 2024-06-26T07:16:54.5388795Z Resolving deltas: 59% (9612/16205) 2024-06-26T07:16:54.5389290Z Resolving deltas: 60% (9871/16205) 2024-06-26T07:16:54.5407787Z Resolving deltas: 61% (9887/16205) 2024-06-26T07:16:54.5422752Z Resolving deltas: 62% (10063/16205) 2024-06-26T07:16:54.5432905Z Resolving deltas: 63% (10265/16205) 2024-06-26T07:16:54.5436516Z Resolving deltas: 64% (10462/16205) 2024-06-26T07:16:54.5445284Z Resolving deltas: 65% (10537/16205) 2024-06-26T07:16:54.5453140Z Resolving deltas: 66% (10723/16205) 2024-06-26T07:16:54.5473063Z Resolving deltas: 67% (10885/16205) 2024-06-26T07:16:54.5496259Z Resolving deltas: 69% (11189/16205) 2024-06-26T07:16:54.5503669Z Resolving deltas: 70% (11378/16205) 2024-06-26T07:16:54.5511385Z Resolving deltas: 71% (11507/16205) 2024-06-26T07:16:54.5525253Z Resolving deltas: 72% (11676/16205) 2024-06-26T07:16:54.5537713Z Resolving deltas: 73% (11861/16205) 2024-06-26T07:16:54.5550667Z Resolving deltas: 74% (11992/16205) 2024-06-26T07:16:54.5585485Z Resolving deltas: 75% (12154/16205) 2024-06-26T07:16:54.5588296Z Resolving deltas: 78% (12721/16205) 2024-06-26T07:16:54.5596357Z Resolving deltas: 79% (12804/16205) 2024-06-26T07:16:54.5603553Z Resolving deltas: 80% (12983/16205) 2024-06-26T07:16:54.5612941Z Resolving deltas: 81% (13132/16205) 2024-06-26T07:16:54.5621568Z Resolving deltas: 82% (13289/16205) 2024-06-26T07:16:54.5627415Z Resolving deltas: 83% (13455/16205) 2024-06-26T07:16:54.5638025Z Resolving deltas: 84% (13615/16205) 2024-06-26T07:16:54.5659411Z Resolving deltas: 85% (13775/16205) 2024-06-26T07:16:54.5667958Z Resolving deltas: 86% (13937/16205) 2024-06-26T07:16:54.5677149Z Resolving deltas: 87% (14099/16205) 2024-06-26T07:16:54.5681463Z Resolving deltas: 88% (14315/16205) 2024-06-26T07:16:54.5689668Z Resolving deltas: 89% (14423/16205) 2024-06-26T07:16:54.5700153Z Resolving deltas: 90% (14590/16205) 2024-06-26T07:16:54.5707611Z Resolving deltas: 91% (14770/16205) 2024-06-26T07:16:54.5717925Z Resolving deltas: 92% (14913/16205) 2024-06-26T07:16:54.5721779Z Resolving deltas: 93% (15141/16205) 2024-06-26T07:16:54.5730328Z Resolving deltas: 94% (15255/16205) 2024-06-26T07:16:54.5737054Z Resolving deltas: 95% (15426/16205) 2024-06-26T07:16:54.5744351Z Resolving deltas: 96% (15561/16205) 2024-06-26T07:16:54.5750800Z Resolving deltas: 97% (15719/16205) 2024-06-26T07:16:54.5759266Z Resolving deltas: 98% (15888/16205) 2024-06-26T07:16:54.5793314Z Resolving deltas: 99% (16048/16205) 2024-06-26T07:16:54.5793758Z Resolving deltas: 100% (16205/16205) 2024-06-26T07:16:54.5794195Z Resolving deltas: 100% (16205/16205), done. 2024-06-26T07:16:55.7935855Z + pushd torchbench 2024-06-26T07:16:55.7936265Z ~/workspace/torchbench ~/workspace 2024-06-26T07:16:55.7936783Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:16:55.8527121Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:16:55.8527702Z 2024-06-26T07:16:55.8528266Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:16:55.8529119Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:16:55.8529878Z state without impacting any branches by switching back to a branch. 2024-06-26T07:16:55.8530333Z 2024-06-26T07:16:55.8530666Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:16:55.8531506Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:16:55.8532035Z 2024-06-26T07:16:55.8532256Z git switch -c 2024-06-26T07:16:55.8532622Z 2024-06-26T07:16:55.8532818Z Or undo this operation with: 2024-06-26T07:16:55.8533163Z 2024-06-26T07:16:55.8533319Z git switch - 2024-06-26T07:16:55.8533550Z 2024-06-26T07:16:55.8533895Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:16:55.8534389Z 2024-06-26T07:16:55.8534570Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:16:55.8535028Z + '[' '' ']' 2024-06-26T07:16:55.8535390Z + python install.py --continue_on_fail 2024-06-26T07:16:57.8255885Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:17:13.5991482Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:17:15.9712386Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:17:18.5448765Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:17:20.9916355Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:17:21.0068468Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:17:29.9169786Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:17:33.2789851Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:17:36.6401617Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:17:40.0238671Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:17:40.0240794Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:17:41.8300959Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:17:45.4620507Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:17:45.4779715Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:18:40.3509136Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:18:49.1136665Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:18:56.7565303Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:19:04.0439305Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:19:13.0273127Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:19:20.4587240Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:19:27.4883174Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:19:35.0777621Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:19:42.7555755Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:19:50.4750233Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:19:57.8182479Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:20:05.3688589Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:20:09.3918461Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:20:51.9440791Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:20:57.9745074Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:21:04.6169167Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:21:15.6259116Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:21:17.8075588Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:21:20.2611742Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:21:28.2568383Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:21:36.4076364Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:21:43.0792061Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:21:51.7005679Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:21:58.7837307Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:22:05.0688817Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:22:12.4284673Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:22:28.7523844Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:22:36.1005695Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:22:41.4351101Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:22:47.5064153Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:22:55.3315558Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:22:55.3320221Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:23:08.6542468Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:23:13.9509799Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:23:13.9512862Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:23:22.5423173Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:23:24.8085931Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:23:27.0961375Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:24:07.2152467Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:25:27.3698533Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:25:27.3856325Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:25:29.9174616Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:25:29.9333595Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:25:29.9490793Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:25:29.9644733Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:25:29.9799692Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:25:29.9952294Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:25:50.5346875Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:25:50.5350888Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:25:52.8239700Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:25:56.2771159Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:25:56.2928945Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:25:56.3085015Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:25:56.3235745Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:25:56.3388722Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:25:56.3541289Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:26:03.5647231Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:26:06.1172438Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:26:10.7161523Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:26:10.7317915Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:26:10.7475739Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:26:10.7629940Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:26:10.7783722Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:26:10.7934378Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:26:20.9922586Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:26:33.0186347Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:26:33.0339045Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:26:33.0340775Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:26:33.0342828Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:26:36.7645916Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:26:39.6820929Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:26:39.6980545Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:26:58.8513550Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:27:05.7013521Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:27:10.0705021Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:27:18.1159574Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:27:18.1320038Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:27:18.1479562Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:27:18.1634659Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:27:18.1787200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:27:18.1936812Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:27:18.2084826Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:27:18.2232118Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:27:29.7464914Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:27:42.4683484Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:27:42.4839843Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:27:45.4896736Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:27:48.6970985Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:27:49.1179748Z + popd 2024-06-26T07:27:49.1180092Z ~/workspace 2024-06-26T07:27:49.1182579Z + [[ dynamic_inductor_torchbench != *cpu_inductor* ]] 2024-06-26T07:27:49.1183116Z + install_torchrec_and_fbgemm 2024-06-26T07:27:49.1183496Z + local torchrec_commit 2024-06-26T07:27:49.1184019Z ++ get_pinned_commit torchrec 2024-06-26T07:27:49.1184492Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:27:49.1263263Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:27:49.1263785Z + local fbgemm_commit 2024-06-26T07:27:49.1265272Z ++ get_pinned_commit fbgemm 2024-06-26T07:27:49.1265687Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:27:49.1273635Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:27:49.1274297Z + pip_uninstall torchrec-nightly 2024-06-26T07:27:49.1274777Z + pip uninstall -y torchrec-nightly 2024-06-26T07:27:49.4846032Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:27:49.5262374Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:27:49.5262946Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:27:49.8721779Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:27:49.9145041Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:27:49.9146003Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:27:50.3484341Z Collecting setuptools-git-versioning 2024-06-26T07:27:50.3735358Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:27:50.4770829Z Collecting scikit-build 2024-06-26T07:27:50.4808315Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:27:50.5777655Z Collecting pyre-extensions 2024-06-26T07:27:50.5813780Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:27:50.6250802Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:27:50.6252908Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:27:50.6800875Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:27:50.6835780Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:27:50.7394886Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:27:50.7405029Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:27:50.7410109Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:27:50.7864516Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:27:50.7900432Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:27:50.8315114Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:27:50.8540686Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:27:50.8593002Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:27:50.9058426Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:27:50.9638448Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:27:51.0158887Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:27:51.0720875Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:27:53.2598675Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:27:53.5792693Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:27:53.6987847Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:27:53.6989117Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:27:53.6991011Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:27:54.0357070Z Collecting fbgemm-gpu 2024-06-26T07:27:54.0360740Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-5c3f2m3v/fbgemm-gpu_0eccf7c44f5b4d3fb3a7ffc90945a4c0 2024-06-26T07:27:54.0375695Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-5c3f2m3v/fbgemm-gpu_0eccf7c44f5b4d3fb3a7ffc90945a4c0 2024-06-26T07:27:54.9865799Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:27:54.9880380Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:27:55.3910243Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:27:55.8319777Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:27:55.8321144Z Running command git submodule update --init --recursive -q 2024-06-26T07:28:03.3726903Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:28:03.3748569Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:28:03.3754915Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:10:07.1102719Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T08:10:07.2905200Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252635452 sha256=7a941307b62e70de4b9f506657acc7c416bc8de9bae661ae521836021571683d 2024-06-26T08:10:07.2906849Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:10:07.2926821Z Successfully built fbgemm-gpu 2024-06-26T08:10:09.4628271Z Installing collected packages: fbgemm-gpu 2024-06-26T08:10:12.6144041Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:10:13.2373690Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:13.2375216Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:13.5758332Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:13.5761746Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-w7rwd3q8 2024-06-26T08:10:13.5779183Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-w7rwd3q8 2024-06-26T08:10:14.2207550Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:10:14.2226246Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:14.4955680Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:14.8094213Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:15.0373358Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:10:15.0422286Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:10:15.0425112Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:10:15.0429567Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:10:15.0927842Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:10:15.1103800Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:10:15.1179941Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:10:15.1183265Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:10:15.1196107Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:10:15.1216425Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:10:15.1873568Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:10:15.1876873Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:10:15.1882014Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:10:15.2655415Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:10:15.2660698Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:10:15.2859011Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:10:15.2894760Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:10:15.3072874Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:10:15.3075590Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:10:15.3109737Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:10:15.3182584Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:10:15.3186463Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:10:15.3189734Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:10:15.3192971Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:10:15.3196103Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:10:15.3870950Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:10:15.4188764Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:10:15.4336709Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:10:15.4562092Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:10:15.4660756Z Building wheels for collected packages: torchrec 2024-06-26T08:10:15.8957397Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:10:15.8967619Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=6fd1f3161da91f71615fe28bc189d208b139cf31c79ebe67fe524100e0602682 2024-06-26T08:10:15.8969273Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:10:15.8990386Z Successfully built torchrec 2024-06-26T08:10:17.9670783Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:10:18.5276301Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:10:18.6654842Z ++ pwd 2024-06-26T08:10:18.6657330Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:10:18.6657929Z + test_dynamo_benchmark torchbench 0 2024-06-26T08:10:18.6661138Z ++ pwd 2024-06-26T08:10:18.6662106Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:10:18.6663332Z + local suite=torchbench 2024-06-26T08:10:18.6663704Z + shift 2024-06-26T08:10:18.6664178Z + local shard_id=0 2024-06-26T08:10:18.6664487Z + shift 2024-06-26T08:10:18.6664934Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-26T08:10:18.6670035Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-26T08:10:18.6670595Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-26T08:10:18.6671161Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-26T08:10:18.6671923Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-06-26T08:10:18.6672676Z ++ pwd 2024-06-26T08:10:18.6673157Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:10:18.6673922Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:10:18.6696069Z + local name=inference 2024-06-26T08:10:18.6696550Z + shift 2024-06-26T08:10:18.6696892Z + local suite=torchbench 2024-06-26T08:10:18.6697246Z + shift 2024-06-26T08:10:18.6697522Z + local shard_id=0 2024-06-26T08:10:18.6697829Z + shift 2024-06-26T08:10:18.6698153Z + partition_flags=() 2024-06-26T08:10:18.6698489Z + local partition_flags 2024-06-26T08:10:18.6698878Z + [[ -n 2 ]] 2024-06-26T08:10:18.6699265Z + [[ -n 0 ]] 2024-06-26T08:10:18.6699861Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:10:18.6700579Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-26T08:10:18.6701110Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-26T08:10:18.6701634Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-26T08:10:18.6703535Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:10:24.3684599Z 2024-06-26T08:10:24.9246270Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-06-26T08:10:24.9248394Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-06-26T08:10:25.1154083Z 2024-06-26T08:10:25.1155193Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:25.1155719Z cuda eval torchrec_dlrm 2024-06-26T08:10:25.1164127Z Traceback (most recent call last): 2024-06-26T08:10:25.1165013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:10:25.1165997Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:10:25.1166857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 447, in forward_pass 2024-06-26T08:10:25.1167685Z return mod(*inputs) 2024-06-26T08:10:25.1168809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:10:25.1169819Z return self._call_impl(*args, **kwargs) 2024-06-26T08:10:25.1170952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:10:25.1171873Z return forward_call(*args, **kwargs) 2024-06-26T08:10:25.1172830Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-06-26T08:10:25.1173820Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-06-26T08:10:25.1175068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:10:25.1176050Z return self._call_impl(*args, **kwargs) 2024-06-26T08:10:25.1177059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:10:25.1177959Z return forward_call(*args, **kwargs) 2024-06-26T08:10:25.1178910Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-06-26T08:10:25.1179790Z concatenated_dense = self.inter_arch( 2024-06-26T08:10:25.1180867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:10:25.1181869Z return self._call_impl(*args, **kwargs) 2024-06-26T08:10:25.1182866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:10:25.1184257Z return forward_call(*args, **kwargs) 2024-06-26T08:10:25.1185371Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-06-26T08:10:25.1186374Z return self.crossnet(combined_values.reshape([B, -1])) 2024-06-26T08:10:25.1187499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:10:25.1188466Z return self._call_impl(*args, **kwargs) 2024-06-26T08:10:25.1189535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:10:25.1190451Z return forward_call(*args, **kwargs) 2024-06-26T08:10:25.1191409Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-06-26T08:10:25.1192422Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-06-26T08:10:25.1193356Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-06-26T08:10:25.1193988Z 2024-06-26T08:10:25.1194301Z The above exception was the direct cause of the following exception: 2024-06-26T08:10:25.1194812Z 2024-06-26T08:10:25.1194972Z Traceback (most recent call last): 2024-06-26T08:10:25.1195690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:10:25.1196421Z ) = runner.load_model( 2024-06-26T08:10:25.1197133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:10:25.1197963Z self.validate_model(model, example_inputs) 2024-06-26T08:10:25.1198782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:10:25.1199614Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:10:25.1200133Z RuntimeError: Eager run failed 2024-06-26T08:10:25.1200403Z 2024-06-26T08:10:25.1200570Z eager_fail_to_run 2024-06-26T08:10:28.4332052Z 2024-06-26T08:10:30.4171343Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:30.4171908Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:10:30.4234772Z cuda eval BERT_pytorch 2024-06-26T08:11:26.5721307Z pass 2024-06-26T08:11:26.5733419Z TIMING: code_gen:5.06504 inductor_compile:16.85766 backend_compile:44.83707 entire_frame_compile:53.88558 2024-06-26T08:11:26.5735036Z STATS: call_* op count: 1100 | FakeTensor.__torch_dispatch__:3212 | FakeTensorMode.__torch_dispatch__:60309 | attempt fast:2870 | fast is_contiguous:2870 | ProxyTorchDispatchMode.__torch_dispatch__:12905 2024-06-26T08:11:26.5736492Z Dynamo produced 1 graphs covering 1100 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:32.6119482Z 2024-06-26T08:11:35.1792118Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:35.1792602Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:11:35.1841197Z cuda eval Background_Matting 2024-06-26T08:11:35.1844369Z pass_due_to_skip 2024-06-26T08:11:35.2363806Z TIMING: 2024-06-26T08:11:35.2364145Z STATS: call_* op count: 0 2024-06-26T08:11:35.2364709Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:38.5325723Z 2024-06-26T08:11:41.2174898Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:41.2175478Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:11:41.2201133Z cuda eval LearningToPaint 2024-06-26T08:11:48.4895434Z pass 2024-06-26T08:11:48.4913841Z TIMING: code_gen:1.64634 inductor_compile:2.86344 backend_compile:5.2322 entire_frame_compile:5.87894 2024-06-26T08:11:48.4915135Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1544 | FakeTensorMode.__torch_dispatch__:11677 | ProxyTorchDispatchMode.__torch_dispatch__:3014 2024-06-26T08:11:48.4916271Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:52.2418505Z 2024-06-26T08:11:53.4799229Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-06-26T08:11:53.4971581Z 2024-06-26T08:11:53.4971868Z 2024-06-26T08:11:53.5975979Z 0% 0.00/528M [00:00 will be ignored 2024-06-26T08:53:50.0953390Z pass 2024-06-26T08:53:50.1208108Z TIMING: entire_frame_compile:195.1056 code_gen:37.3096 inductor_compile:151.1177 backend_compile:180.95805 2024-06-26T08:53:50.1210134Z STATS: call_* op count: 672 | FakeTensorMode.__torch_dispatch__:82541 | FakeTensor.__torch_dispatch__:11106 | attempt fast:2315 | fast is_contiguous:2089 | slow no contiguity match:226 | ProxyTorchDispatchMode.__torch_dispatch__:17164 2024-06-26T08:53:50.1211929Z Dynamo produced 3 graphs covering 672 ops with 7 graph breaks (5 unique) 2024-06-26T08:53:58.2969767Z 2024-06-26T08:53:59.5340683Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:53:59.5341157Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:53:59.5341585Z cuda train alexnet 2024-06-26T08:54:17.7920694Z pass 2024-06-26T08:54:17.8124986Z TIMING: entire_frame_compile:11.98096 code_gen:7.87192 inductor_compile:9.86138 backend_compile:11.46528 2024-06-26T08:54:17.8126595Z STATS: call_* op count: 26 | FakeTensor.__torch_dispatch__:441 | FakeTensorMode.__torch_dispatch__:4251 | attempt fast:34 | fast is_contiguous:34 | ProxyTorchDispatchMode.__torch_dispatch__:950 2024-06-26T08:54:17.8127896Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2024-06-26T08:54:21.5453984Z 2024-06-26T08:54:23.2386076Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:23.2386693Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:54:23.2387174Z cuda train basic_gnn_edgecnn 2024-06-26T08:54:26.3977744Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:54:28.2598754Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:54:28.2599551Z File "/tmp/jenkins_pyg/tmpiyky970b.py", line 190, in propagate 2024-06-26T08:54:28.2601550Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-06-26T08:54:28.2602424Z File "/tmp/jenkins_pyg/tmpiyky970b.py", line 128, in _collect 2024-06-26T08:54:28.2602995Z x_j = self._lift(x_j, edge_def, j) 2024-06-26T08:54:28.2603562Z File "/tmp/jenkins_pyg/tmpiyky970b.py", line 88, in _lift 2024-06-26T08:54:28.2604145Z return src.index_select(self.node_dim, index) 2024-06-26T08:54:28.2604482Z 2024-06-26T08:54:28.2717753Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:54:28.2718609Z File "/tmp/jenkins_pyg/tmpiyky970b.py", line 190, in propagate 2024-06-26T08:54:28.2719365Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-06-26T08:54:28.2720226Z File "/tmp/jenkins_pyg/tmpiyky970b.py", line 128, in _collect 2024-06-26T08:54:28.2721002Z x_j = self._lift(x_j, edge_def, j) 2024-06-26T08:54:28.2721701Z File "/tmp/jenkins_pyg/tmpiyky970b.py", line 88, in _lift 2024-06-26T08:54:28.2722416Z return src.index_select(self.node_dim, index) 2024-06-26T08:54:28.2722768Z 2024-06-26T08:54:28.3960127Z W0626 08:54:28.395000 140337282466432 torch/_logging/_internal.py:1040] [24/0] Profiler function will be ignored 2024-06-26T08:54:33.6080083Z pass 2024-06-26T08:54:33.6090252Z TIMING: entire_frame_compile:7.59682 inductor_compile:5.10042 backend_compile:5.87547 code_gen:3.63608 2024-06-26T08:54:33.6091556Z STATS: call_* op count: 103 | FakeTensorMode.__torch_dispatch__:7254 | ProxyTorchDispatchMode.__torch_dispatch__:1787 | FakeTensor.__torch_dispatch__:1007 2024-06-26T08:54:33.6092704Z Dynamo produced 8 graphs covering 103 ops with 20 graph breaks (6 unique) 2024-06-26T08:54:37.3452157Z 2024-06-26T08:54:39.1030948Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:39.1031458Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:54:39.1031954Z cuda train basic_gnn_gcn 2024-06-26T08:54:41.6144308Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:54:42.1358678Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:54:42.5060328Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:54:42.7378917Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:54:43.2139524Z W0626 08:54:43.213000 140297506611840 torch/_logging/_internal.py:1040] [18/0] Profiler function will be ignored 2024-06-26T08:54:46.7586945Z pass 2024-06-26T08:54:46.7591101Z TIMING: entire_frame_compile:5.27522 inductor_compile:3.6999 backend_compile:4.34464 code_gen:2.63455 2024-06-26T08:54:46.7592610Z STATS: call_* op count: 99 | FakeTensorMode.__torch_dispatch__:4291 | FakeTensor.__torch_dispatch__:551 | ProxyTorchDispatchMode.__torch_dispatch__:1004 2024-06-26T08:54:46.7593857Z Dynamo produced 10 graphs covering 99 ops with 13 graph breaks (6 unique) 2024-06-26T08:54:50.3861831Z 2024-06-26T08:54:52.1216917Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:52.1217507Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:54:52.1217961Z cuda train basic_gnn_gin 2024-06-26T08:54:54.9738418Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:54:55.7907136Z W0626 08:54:55.789000 140083812250240 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:55:00.1931671Z pass 2024-06-26T08:55:00.1970738Z TIMING: entire_frame_compile:6.45524 code_gen:3.10597 inductor_compile:4.63217 backend_compile:5.35125 2024-06-26T08:55:00.1972014Z STATS: call_* op count: 113 | FakeTensorMode.__torch_dispatch__:6784 | FakeTensor.__torch_dispatch__:965 | ProxyTorchDispatchMode.__torch_dispatch__:1747 2024-06-26T08:55:00.1973162Z Dynamo produced 3 graphs covering 113 ops with 7 graph breaks (5 unique) 2024-06-26T08:55:03.8682874Z 2024-06-26T08:55:05.6102108Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:55:05.6102741Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:55:05.6103706Z cuda train basic_gnn_sage 2024-06-26T08:55:08.2880475Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:55:08.8980787Z W0626 08:55:08.897000 140494083707520 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:55:12.5735565Z pass 2024-06-26T08:55:12.5760568Z TIMING: entire_frame_compile:5.55099 code_gen:2.65537 inductor_compile:3.93373 backend_compile:4.61533 2024-06-26T08:55:12.5762107Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:5526 | FakeTensor.__torch_dispatch__:739 | ProxyTorchDispatchMode.__torch_dispatch__:1396 2024-06-26T08:55:12.5763233Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique) 2024-06-26T08:55:16.2076554Z 2024-06-26T08:55:16.6946755Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:55:16.6947437Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:55:16.6947962Z cuda train dcgan 2024-06-26T08:55:23.4429975Z pass 2024-06-26T08:55:23.4448298Z TIMING: entire_frame_compile:5.26802 code_gen:1.21051 inductor_compile:3.03935 backend_compile:4.64423 2024-06-26T08:55:23.4449769Z STATS: call_* op count: 17 | FakeTensor.__torch_dispatch__:534 | FakeTensorMode.__torch_dispatch__:3711 | attempt fast:121 | fast is_contiguous:121 | ProxyTorchDispatchMode.__torch_dispatch__:842 2024-06-26T08:55:23.4451070Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2024-06-26T08:55:27.0687322Z 2024-06-26T08:55:30.4541539Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:55:30.4542535Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:55:30.4543400Z cuda train demucs 2024-06-26T08:56:27.9704503Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Error while creating guard: 2024-06-26T08:56:27.9708728Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Name: '' 2024-06-26T08:56:27.9710021Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Source: shape_env 2024-06-26T08:56:27.9711019Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Create Function: SHAPE_ENV 2024-06-26T08:56:27.9711984Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Guard Types: None 2024-06-26T08:56:27.9712844Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Code List: None 2024-06-26T08:56:27.9713724Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Object Weakref: None 2024-06-26T08:56:27.9714675Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Guarded Class Weakref: None 2024-06-26T08:56:27.9715660Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] Traceback (most recent call last): 2024-06-26T08:56:27.9717369Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 259, in create 2024-06-26T08:56:27.9718815Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] return self.create_fn(builder, self) 2024-06-26T08:56:27.9720486Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1680, in SHAPE_ENV 2024-06-26T08:56:27.9722041Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] guards = output_graph.shape_env.produce_guards( 2024-06-26T08:56:27.9723754Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4180, in produce_guards 2024-06-26T08:56:27.9725253Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] raise ConstraintViolationError( 2024-06-26T08:56:27.9727144Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['streams'].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-06-26T08:56:27.9729392Z E0626 08:56:27.968000 140673882288768 torch/_guards.py:261] [4/0_1] - Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4). 2024-06-26T08:56:27.9730750Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] Created at: 2024-06-26T08:56:27.9732163Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 564, in transform 2024-06-26T08:56:27.9733571Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] tracer = InstructionTranslator( 2024-06-26T08:56:27.9735126Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2385, in __init__ 2024-06-26T08:56:27.9736468Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] output=OutputGraph( 2024-06-26T08:56:27.9737942Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 313, in __init__ 2024-06-26T08:56:27.9739294Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] self.init_ambient_guards() 2024-06-26T08:56:27.9740848Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 452, in init_ambient_guards 2024-06-26T08:56:27.9742463Z E0626 08:56:27.970000 140673882288768 torch/_guards.py:263] [4/0_1] self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV)) 2024-06-26T08:56:27.9743318Z ERROR:common: 2024-06-26T08:56:27.9743655Z Traceback (most recent call last): 2024-06-26T08:56:27.9744842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2627, in check_accuracy 2024-06-26T08:56:27.9745714Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2024-06-26T08:56:27.9746713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 434, in _fn 2024-06-26T08:56:27.9747536Z return fn(*args, **kwargs) 2024-06-26T08:56:27.9748249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T08:56:27.9749087Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T08:56:27.9750008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 450, in forward_and_backward_pass 2024-06-26T08:56:27.9750846Z cloned_inputs = clone_inputs(inputs) 2024-06-26T08:56:27.9751972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 451, in torch_dynamo_resume_in_forward_and_backward_pass_at_450 2024-06-26T08:56:27.9753013Z self.optimizer_zero_grad(mod) 2024-06-26T08:56:27.9753978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:56:27.9754933Z pred = mod(*cloned_inputs) 2024-06-26T08:56:27.9755879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:56:27.9756790Z return self._call_impl(*args, **kwargs) 2024-06-26T08:56:27.9757727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:56:27.9758563Z return forward_call(*args, **kwargs) 2024-06-26T08:56:27.9759487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1121, in __call__ 2024-06-26T08:56:27.9760354Z return self._torchdynamo_orig_callable( 2024-06-26T08:56:27.9761290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 948, in __call__ 2024-06-26T08:56:27.9762115Z result = self._inner_convert( 2024-06-26T08:56:27.9763019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 472, in __call__ 2024-06-26T08:56:27.9763831Z return _compile( 2024-06-26T08:56:27.9764670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 85, in wrapper_function 2024-06-26T08:56:27.9765806Z return StrobelightCompileTimeProfiler.profile_compile_time( 2024-06-26T08:56:27.9766999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time 2024-06-26T08:56:27.9767949Z return func(*args, **kwargs) 2024-06-26T08:56:27.9768559Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T08:56:27.9769207Z return func(*args, **kwds) 2024-06-26T08:56:27.9770068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 817, in _compile 2024-06-26T08:56:27.9771031Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-06-26T08:56:27.9772063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T08:56:27.9772868Z r = func(*args, **kwargs) 2024-06-26T08:56:27.9773760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 726, in compile_inner 2024-06-26T08:56:27.9774630Z check_fn = CheckFunctionManager( 2024-06-26T08:56:27.9775504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2094, in __init__ 2024-06-26T08:56:27.9776289Z guard.create(builder) 2024-06-26T08:56:27.9777031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 259, in create 2024-06-26T08:56:27.9777801Z return self.create_fn(builder, self) 2024-06-26T08:56:27.9778693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1680, in SHAPE_ENV 2024-06-26T08:56:27.9779554Z guards = output_graph.shape_env.produce_guards( 2024-06-26T08:56:27.9780648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4180, in produce_guards 2024-06-26T08:56:27.9781618Z raise ConstraintViolationError( 2024-06-26T08:56:27.9782810Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['streams'].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-06-26T08:56:27.9784637Z - Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4). 2024-06-26T08:56:27.9785421Z 2024-06-26T08:56:27.9785426Z 2024-06-26T08:56:27.9785828Z You can suppress this exception and fall back to eager by setting: 2024-06-26T08:56:27.9786407Z import torch._dynamo 2024-06-26T08:56:27.9786893Z torch._dynamo.config.suppress_errors = True 2024-06-26T08:56:27.9787250Z 2024-06-26T08:56:27.9787561Z TorchDynamo optimized model failed to run because of following error 2024-06-26T08:56:27.9788147Z fail_to_run 2024-06-26T08:56:28.0056461Z TIMING: entire_frame_compile:0.50123 inductor_compile:1.35131 backend_compile:1.81961 code_gen:0.44562 2024-06-26T08:56:28.0058272Z STATS: call_* op count: 34 | FakeTensorMode.__torch_dispatch__:1562 | attempt fast:30 | fast is_contiguous:12 | slow no contiguity match:12 | slow both tensors nontrivially broadcast:6 | FakeTensor.__torch_dispatch__:199 | ProxyTorchDispatchMode.__torch_dispatch__:140 2024-06-26T08:56:28.0059923Z Dynamo produced 2 graphs covering 34 ops with 4 graph breaks (3 unique) 2024-06-26T08:56:31.5696825Z 2024-06-26T08:56:33.6474140Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:56:33.6475035Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:56:33.6475950Z cuda train densenet121 2024-06-26T08:59:31.1176848Z pass 2024-06-26T08:59:31.1248248Z TIMING: entire_frame_compile:164.90491 code_gen:53.97512 inductor_compile:103.95996 backend_compile:154.64616 2024-06-26T08:59:31.1250839Z STATS: call_* op count: 435 | FakeTensor.__torch_dispatch__:17439 | FakeTensorMode.__torch_dispatch__:130352 | attempt fast:4230 | fast is_contiguous:3168 | ProxyTorchDispatchMode.__torch_dispatch__:30835 | slow no contiguity match:1062 2024-06-26T08:59:31.1252354Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-06-26T08:59:42.0146396Z 2024-06-26T08:59:46.4796675Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:59:46.4797336Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:59:46.4797971Z cuda train detectron2_maskrcnn_r_50_c4 2024-06-26T08:59:46.4810662Z Traceback (most recent call last): 2024-06-26T08:59:46.4811543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:59:46.4812413Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:59:46.4813355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:59:46.4814256Z pred = mod(*cloned_inputs) 2024-06-26T08:59:46.4815544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:59:46.4816562Z return self._call_impl(*args, **kwargs) 2024-06-26T08:59:46.4817762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:59:46.4818696Z return forward_call(*args, **kwargs) 2024-06-26T08:59:46.4819932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-06-26T08:59:46.4820889Z return self.inference(batched_inputs) 2024-06-26T08:59:46.4822091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2024-06-26T08:59:46.4823403Z results, _ = self.roi_heads(images, features, proposals, None) 2024-06-26T08:59:46.4824823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:59:46.4825726Z return self._call_impl(*args, **kwargs) 2024-06-26T08:59:46.4826656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:59:46.4827488Z return forward_call(*args, **kwargs) 2024-06-26T08:59:46.4828524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2024-06-26T08:59:46.4829444Z box_features = self._shared_roi_transform( 2024-06-26T08:59:46.4830545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2024-06-26T08:59:46.4831703Z x = self.pooler(features, boxes) 2024-06-26T08:59:46.4832778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:59:46.4833683Z return self._call_impl(*args, **kwargs) 2024-06-26T08:59:46.4834609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:59:46.4835448Z return forward_call(*args, **kwargs) 2024-06-26T08:59:46.4836353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2024-06-26T08:59:46.4837251Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2024-06-26T08:59:46.4838284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:59:46.4839177Z return self._call_impl(*args, **kwargs) 2024-06-26T08:59:46.4840107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:59:46.4840952Z return forward_call(*args, **kwargs) 2024-06-26T08:59:46.4841850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2024-06-26T08:59:46.4842648Z return roi_align( 2024-06-26T08:59:46.4843485Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T08:59:46.4844596Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T08:59:46.4845906Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T08:59:46.4847003Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T08:59:46.4848238Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T08:59:46.4849120Z v1 = masked_index(y_low, x_low) 2024-06-26T08:59:46.4850026Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T08:59:46.4850840Z return input[ 2024-06-26T08:59:46.4853935Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 11110 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:59:46.4856692Z 2024-06-26T08:59:46.4856994Z The above exception was the direct cause of the following exception: 2024-06-26T08:59:46.4857457Z 2024-06-26T08:59:46.4857608Z Traceback (most recent call last): 2024-06-26T08:59:46.4858302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:59:46.4858986Z ) = runner.load_model( 2024-06-26T08:59:46.4859660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:59:46.4860434Z self.validate_model(model, example_inputs) 2024-06-26T08:59:46.4861198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:59:46.4861966Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:59:46.4862445Z RuntimeError: Eager run failed 2024-06-26T08:59:46.4862709Z 2024-06-26T08:59:46.4867214Z eager_fail_to_run 2024-06-26T08:59:49.7967234Z 2024-06-26T08:59:57.9429679Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:59:57.9430264Z loading model: 0it [00:08, ?it/s] 2024-06-26T08:59:57.9430766Z cuda train dlrm 2024-06-26T09:00:06.1529378Z skipping cudagraphs due to sparsity not handled. Please file issue for sparse inference weights. Found from : 2024-06-26T09:00:06.1531011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:00:06.1531986Z pred = mod(*cloned_inputs) 2024-06-26T09:00:06.1532995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:00:06.1533843Z return forward_call(*args, **kwargs) 2024-06-26T09:00:06.1534725Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 355, in forward 2024-06-26T09:00:06.1535678Z return self.sequential_forward(dense_x, lS_o, lS_i) 2024-06-26T09:00:06.1536693Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 367, in sequential_forward 2024-06-26T09:00:06.1537658Z ly = self.apply_emb(lS_o, lS_i, self.emb_l) 2024-06-26T09:00:06.1538581Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 308, in apply_emb 2024-06-26T09:00:06.1539564Z V = E(sparse_index_group_batch, sparse_offset_group_batch) 2024-06-26T09:00:06.1539958Z 2024-06-26T09:00:06.6219050Z pass 2024-06-26T09:00:06.6630060Z TIMING: entire_frame_compile:5.94116 code_gen:1.29612 inductor_compile:1.94857 backend_compile:5.53901 2024-06-26T09:00:06.6631385Z STATS: call_* op count: 40 | FakeTensor.__torch_dispatch__:548 | FakeTensorMode.__torch_dispatch__:5142 | ProxyTorchDispatchMode.__torch_dispatch__:1507 2024-06-26T09:00:06.6632502Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2024-06-26T09:00:10.2602679Z 2024-06-26T09:00:11.6273235Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:00:11.6273830Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:00:11.6274364Z cuda train drq 2024-06-26T09:00:19.0734790Z W0626 09:00:19.072000 139826704704128 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:00:28.4683314Z pass 2024-06-26T09:00:28.4712952Z TIMING: entire_frame_compile:14.01097 code_gen:6.94816 inductor_compile:9.1084 backend_compile:12.23632 2024-06-26T09:00:28.4714275Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:12454 | FakeTensor.__torch_dispatch__:1952 | ProxyTorchDispatchMode.__torch_dispatch__:3300 2024-06-26T09:00:28.4715407Z Dynamo produced 5 graphs covering 137 ops with 7 graph breaks (5 unique) 2024-06-26T09:00:32.4678194Z 2024-06-26T09:00:34.0696166Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:00:34.0696783Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:00:34.0703640Z cuda train fastNLP_Bert 2024-06-26T09:00:40.5396774Z W0626 09:00:40.539000 140211067187840 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T09:00:40.5398251Z W0626 09:00:40.539000 140211067187840 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T09:00:40.5399431Z W0626 09:00:40.539000 140211067187840 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T09:00:40.5400606Z W0626 09:00:40.539000 140211067187840 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T09:00:40.5401946Z W0626 09:00:40.539000 140211067187840 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T09:00:40.5403124Z W0626 09:00:40.539000 140211067187840 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T09:01:16.3353523Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:01:16.3355044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward 2024-06-26T09:01:16.3357539Z embedding_output = self.embeddings(input_ids, token_type_ids) 2024-06-26T09:01:16.3359134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:01:16.3360198Z return forward_call(*args, **kwargs) 2024-06-26T09:01:16.3361143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 236, in forward 2024-06-26T09:01:16.3362063Z words_embeddings = self.word_embeddings(input_ids) 2024-06-26T09:01:16.3362438Z 2024-06-26T09:01:19.3956003Z pass 2024-06-26T09:01:19.4589628Z TIMING: entire_frame_compile:24.98204 inductor_compile:20.18038 backend_compile:21.48584 code_gen:11.59952 2024-06-26T09:01:19.4590958Z STATS: call_* op count: 450 | FakeTensorMode.__torch_dispatch__:53395 | ProxyTorchDispatchMode.__torch_dispatch__:19106 | FakeTensor.__torch_dispatch__:6560 2024-06-26T09:01:19.4592128Z Dynamo produced 7 graphs covering 450 ops with 10 graph breaks (6 unique) 2024-06-26T09:01:24.2851984Z 2024-06-26T09:01:25.0682952Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:01:25.0683431Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:01:25.0683908Z cuda train functorch_dp_cifar10 2024-06-26T09:02:02.2355922Z W0626 09:02:02.234000 139712378049152 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:02:22.2768631Z pass 2024-06-26T09:02:22.2817552Z TIMING: entire_frame_compile:53.43533 code_gen:16.5206 inductor_compile:32.69023 backend_compile:48.02027 2024-06-26T09:02:22.2820250Z STATS: call_* op count: 277 | FakeTensor.__torch_dispatch__:7221 | FakeTensorMode.__torch_dispatch__:42828 | attempt fast:1440 | fast is_contiguous:1400 | ProxyTorchDispatchMode.__torch_dispatch__:7938 | slow both tensors nontrivially broadcast:40 2024-06-26T09:02:22.2821832Z Dynamo produced 3 graphs covering 277 ops with 7 graph breaks (5 unique) 2024-06-26T09:02:28.0226855Z 2024-06-26T09:02:28.5487018Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:28.5487560Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:28.5488065Z cuda train functorch_maml_omniglot 2024-06-26T09:02:35.3659193Z W0626 09:02:35.365000 139874577650304 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:02:41.1737901Z pass 2024-06-26T09:02:41.1738684Z TIMING: entire_frame_compile:8.37511 code_gen:6.0286 inductor_compile:7.48091 backend_compile:7.55834 2024-06-26T09:02:41.1740179Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:1297 | FakeTensorMode.__torch_dispatch__:7066 | ProxyTorchDispatchMode.__torch_dispatch__:1433 2024-06-26T09:02:41.1741670Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T09:02:44.9116467Z 2024-06-26T09:02:46.1089810Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:46.1090339Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:02:46.1090783Z cuda train hf_Albert 2024-06-26T09:04:07.1668305Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:04:07.1671992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:04:07.1673402Z pred = mod(*cloned_inputs) 2024-06-26T09:04:07.1674849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:04:07.1676088Z return forward_call(*args, **kwargs) 2024-06-26T09:04:07.1677591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2024-06-26T09:04:07.1678931Z outputs = self.albert( 2024-06-26T09:04:07.1680191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:04:07.1681488Z return forward_call(*args, **kwargs) 2024-06-26T09:04:07.1683093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-06-26T09:04:07.1684834Z embedding_output = self.embeddings( 2024-06-26T09:04:07.1686628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:04:07.1687949Z return forward_call(*args, **kwargs) 2024-06-26T09:04:07.1689532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-06-26T09:04:07.1691049Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T09:04:07.1691602Z 2024-06-26T09:04:08.6237311Z W0626 09:04:08.622000 140078746100352 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:04:20.3365394Z pass 2024-06-26T09:04:20.3513765Z TIMING: entire_frame_compile:83.51286 code_gen:16.01817 inductor_compile:42.8299 backend_compile:73.94926 2024-06-26T09:04:20.3515294Z STATS: call_* op count: 616 | FakeTensorMode.__torch_dispatch__:82092 | attempt fast:3432 | fast is_contiguous:3432 | FakeTensor.__torch_dispatch__:9135 | ProxyTorchDispatchMode.__torch_dispatch__:20685 2024-06-26T09:04:20.3516641Z Dynamo produced 3 graphs covering 616 ops with 6 graph breaks (5 unique) 2024-06-26T09:04:27.8681984Z 2024-06-26T09:04:31.5870637Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:04:31.5871684Z loading model: 0it [00:03, ?it/s] 2024-06-26T09:04:31.5872707Z cuda train hf_Bart 2024-06-26T09:06:02.2422298Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:06:02.2424056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:06:02.2425281Z pred = mod(*cloned_inputs) 2024-06-26T09:06:02.2428433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:06:02.2429429Z return forward_call(*args, **kwargs) 2024-06-26T09:06:02.2430630Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-06-26T09:06:02.2431919Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-06-26T09:06:02.2433317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:06:02.2434272Z return forward_call(*args, **kwargs) 2024-06-26T09:06:02.2435286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1731, in forward 2024-06-26T09:06:02.2436191Z outputs = self.model( 2024-06-26T09:06:02.2437054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:06:02.2437894Z return forward_call(*args, **kwargs) 2024-06-26T09:06:02.2438892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1599, in forward 2024-06-26T09:06:02.2439795Z encoder_outputs = self.encoder( 2024-06-26T09:06:02.2440704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:06:02.2441548Z return forward_call(*args, **kwargs) 2024-06-26T09:06:02.2442545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1152, in forward 2024-06-26T09:06:02.2443559Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-06-26T09:06:02.2443990Z 2024-06-26T09:06:04.0731704Z W0626 09:06:04.072000 139985660686976 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:07:09.9076750Z pass 2024-06-26T09:07:09.9523905Z TIMING: entire_frame_compile:142.38348 code_gen:31.83966 inductor_compile:74.44321 backend_compile:123.54576 2024-06-26T09:07:09.9526070Z STATS: call_* op count: 1290 | FakeTensorMode.__torch_dispatch__:154553 | FakeTensor.__torch_dispatch__:22375 | attempt fast:3422 | fast is_contiguous:3413 | slow no contiguity match:9 | ProxyTorchDispatchMode.__torch_dispatch__:32320 2024-06-26T09:07:09.9527897Z Dynamo produced 3 graphs covering 1290 ops with 6 graph breaks (5 unique) 2024-06-26T09:07:20.5392177Z 2024-06-26T09:07:22.9465328Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:07:22.9466214Z loading model: 0it [00:02, ?it/s] 2024-06-26T09:07:22.9467080Z cuda train hf_Bert 2024-06-26T09:08:45.1929671Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:08:45.1930900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:08:45.1934253Z pred = mod(*cloned_inputs) 2024-06-26T09:08:45.1935568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:08:45.1936665Z return forward_call(*args, **kwargs) 2024-06-26T09:08:45.1937710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-06-26T09:08:45.1938612Z outputs = self.bert( 2024-06-26T09:08:45.1939645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:08:45.1940642Z return forward_call(*args, **kwargs) 2024-06-26T09:08:45.1941842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T09:08:45.1943057Z embedding_output = self.embeddings( 2024-06-26T09:08:45.1944343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:08:45.1945198Z return forward_call(*args, **kwargs) 2024-06-26T09:08:45.1946212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T09:08:45.1947169Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T09:08:45.1947532Z 2024-06-26T09:08:47.1462386Z W0626 09:08:47.145000 140220388000384 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:09:37.5254608Z pass 2024-06-26T09:09:37.5646276Z TIMING: entire_frame_compile:121.48072 code_gen:25.60735 inductor_compile:61.54432 backend_compile:103.33915 2024-06-26T09:09:37.5649001Z STATS: call_* op count: 1093 | FakeTensorMode.__torch_dispatch__:128404 | attempt fast:3155 | fast is_contiguous:3155 | FakeTensor.__torch_dispatch__:17791 | ProxyTorchDispatchMode.__torch_dispatch__:26943 2024-06-26T09:09:37.5650552Z Dynamo produced 3 graphs covering 1093 ops with 6 graph breaks (5 unique) 2024-06-26T09:09:47.1365280Z 2024-06-26T09:09:51.6212892Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:09:51.6213355Z loading model: 0it [00:04, ?it/s] 2024-06-26T09:09:51.6213802Z cuda train hf_Bert_large 2024-06-26T09:12:35.7750140Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:12:35.7752251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:12:35.7753270Z pred = mod(*cloned_inputs) 2024-06-26T09:12:35.7754292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:12:35.7755145Z return forward_call(*args, **kwargs) 2024-06-26T09:12:35.7756157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-06-26T09:12:35.7757203Z outputs = self.bert( 2024-06-26T09:12:35.7758154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:12:35.7759088Z return forward_call(*args, **kwargs) 2024-06-26T09:12:35.7760507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T09:12:35.7761738Z embedding_output = self.embeddings( 2024-06-26T09:12:35.7762688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:12:35.7763542Z return forward_call(*args, **kwargs) 2024-06-26T09:12:35.7764521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T09:12:35.7765467Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T09:12:35.7765826Z 2024-06-26T09:12:39.3269587Z W0626 09:12:39.326000 140417223631488 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:14:15.4085883Z pass 2024-06-26T09:14:15.5225731Z TIMING: entire_frame_compile:234.1101 code_gen:44.24839 inductor_compile:113.47192 backend_compile:198.68934 2024-06-26T09:14:15.5229678Z STATS: call_* op count: 2113 | FakeTensorMode.__torch_dispatch__:250687 | attempt fast:6107 | fast is_contiguous:6107 | FakeTensor.__torch_dispatch__:34627 | ProxyTorchDispatchMode.__torch_dispatch__:52755 2024-06-26T09:14:15.5231817Z Dynamo produced 3 graphs covering 2113 ops with 6 graph breaks (5 unique) 2024-06-26T09:14:31.2842625Z 2024-06-26T09:14:34.6108403Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:14:34.6108899Z loading model: 0it [00:03, ?it/s] 2024-06-26T09:14:34.6109406Z cuda train hf_BigBird 2024-06-26T09:14:45.8710414Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2024-06-26T09:15:31.3738914Z ERROR:common: 2024-06-26T09:15:31.3739326Z Traceback (most recent call last): 2024-06-26T09:15:31.3740395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 582, in transform 2024-06-26T09:15:31.3741223Z tracer.run() 2024-06-26T09:15:31.3742049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2476, in run 2024-06-26T09:15:31.3742863Z super().run() 2024-06-26T09:15:31.3743785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T09:15:31.3744998Z while self.step(): 2024-06-26T09:15:31.3745898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T09:15:31.3746792Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T09:15:31.3747759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 508, in wrapper 2024-06-26T09:15:31.3748697Z return handle_graph_break(self, inst, speculation.reason) 2024-06-26T09:15:31.3749787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 577, in handle_graph_break 2024-06-26T09:15:31.3750755Z self.output.compile_subgraph(self, reason=reason) 2024-06-26T09:15:31.3751852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1127, in compile_subgraph 2024-06-26T09:15:31.3752873Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2024-06-26T09:15:31.3753669Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T09:15:31.3754310Z return func(*args, **kwds) 2024-06-26T09:15:31.3755278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1324, in compile_and_call_fx_graph 2024-06-26T09:15:31.3756219Z compiled_fn = self.call_user_compiler(gm) 2024-06-26T09:15:31.3757131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T09:15:31.3757921Z r = func(*args, **kwargs) 2024-06-26T09:15:31.3758827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1415, in call_user_compiler 2024-06-26T09:15:31.3760048Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2024-06-26T09:15:31.3761255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1396, in call_user_compiler 2024-06-26T09:15:31.3762210Z compiled_fn = compiler_fn(gm, self.example_inputs()) 2024-06-26T09:15:31.3763214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 129, in __call__ 2024-06-26T09:15:31.3764113Z compiled_gm = compiler_fn(gm, example_inputs) 2024-06-26T09:15:31.3764994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/__init__.py", line 2141, in __call__ 2024-06-26T09:15:31.3765984Z return compile_fx(model_, inputs_, config_patches=self.config) 2024-06-26T09:15:31.3766753Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T09:15:31.3767384Z return func(*args, **kwds) 2024-06-26T09:15:31.3768267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1550, in compile_fx 2024-06-26T09:15:31.3769092Z return aot_autograd( 2024-06-26T09:15:31.3769937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 69, in __call__ 2024-06-26T09:15:31.3770873Z cg = aot_module_simplified(gm, example_inputs, **self.kwargs) 2024-06-26T09:15:31.3771980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 990, in aot_module_simplified 2024-06-26T09:15:31.3772898Z compiled_fn = dispatch_and_compile() 2024-06-26T09:15:31.3773898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 979, in dispatch_and_compile 2024-06-26T09:15:31.3774905Z compiled_fn, _ = create_aot_dispatcher_function( 2024-06-26T09:15:31.3775843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T09:15:31.3776636Z r = func(*args, **kwargs) 2024-06-26T09:15:31.3777636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 708, in create_aot_dispatcher_function 2024-06-26T09:15:31.3778599Z compiled_fn, fw_metadata = compiler_fn( 2024-06-26T09:15:31.3779761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 315, in aot_dispatch_autograd 2024-06-26T09:15:31.3780847Z fw_module, bw_module = aot_config.partition_fn( 2024-06-26T09:15:31.3781850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1482, in partition_fn 2024-06-26T09:15:31.3782739Z _recursive_joint_graph_passes(graph) 2024-06-26T09:15:31.3783763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 269, in _recursive_joint_graph_passes 2024-06-26T09:15:31.3784806Z joint_graph_passes(gm) 2024-06-26T09:15:31.3785773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/joint_graph.py", line 329, in joint_graph_passes 2024-06-26T09:15:31.3786701Z constant_fold_uniform_value(graph) 2024-06-26T09:15:31.3787339Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T09:15:31.3787963Z return func(*args, **kwds) 2024-06-26T09:15:31.3788983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/joint_graph.py", line 229, in constant_fold_uniform_value 2024-06-26T09:15:31.3789922Z cf.run() 2024-06-26T09:15:31.3790712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py", line 200, in run 2024-06-26T09:15:31.3791528Z return super().run(initial_env=env) 2024-06-26T09:15:31.3792370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 146, in run 2024-06-26T09:15:31.3793144Z self.env[node] = self.run_node(node) 2024-06-26T09:15:31.3794190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py", line 162, in run_node 2024-06-26T09:15:31.3795095Z out = super().run_node(node) 2024-06-26T09:15:31.3795944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 203, in run_node 2024-06-26T09:15:31.3796810Z return getattr(self, n.op)(n.target, args, kwargs) 2024-06-26T09:15:31.3797782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 275, in call_function 2024-06-26T09:15:31.3798600Z return target(*args, **kwargs) 2024-06-26T09:15:31.3799375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 670, in __call__ 2024-06-26T09:15:31.3800117Z return self_._op(*args, **kwargs) 2024-06-26T09:15:31.3800762Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2024-06-26T09:15:31.3804084Z OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 14665 has 21.96 GiB memory in use. Of the allocated memory 20.82 GiB is allocated by PyTorch, and 802.08 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T09:15:31.3806824Z 2024-06-26T09:15:31.3807774Z While executing %full_1658 : [num_users=0] = call_function[target=torch.ops.aten.full.default](args = ([4, 12, 16, 64, 64], 0), kwargs = {dtype: torch.float16, layout: torch.strided, device: cuda:0, pin_memory: False}) 2024-06-26T09:15:31.3808972Z Original traceback: 2024-06-26T09:15:31.3809866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:15:31.3810827Z pred = mod(*cloned_inputs) 2024-06-26T09:15:31.3811823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward 2024-06-26T09:15:31.3812755Z outputs = self.bert( 2024-06-26T09:15:31.3813706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward 2024-06-26T09:15:31.3814620Z encoder_outputs = self.encoder( 2024-06-26T09:15:31.3815634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward 2024-06-26T09:15:31.3816563Z layer_outputs = layer_module( 2024-06-26T09:15:31.3817565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward 2024-06-26T09:15:31.3818510Z self_attention_outputs = self.attention( 2024-06-26T09:15:31.3819572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward 2024-06-26T09:15:31.3820485Z self_outputs = self.self( 2024-06-26T09:15:31.3821467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward 2024-06-26T09:15:31.3822542Z context_layer, attention_probs = self.bigbird_block_sparse_attention( 2024-06-26T09:15:31.3823981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 826, in bigbird_block_sparse_attention 2024-06-26T09:15:31.3825345Z last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -1], key_layer, ndim=4) 2024-06-26T09:15:31.3825901Z 2024-06-26T09:15:31.3825907Z 2024-06-26T09:15:31.3826203Z During handling of the above exception, another exception occurred: 2024-06-26T09:15:31.3826651Z 2024-06-26T09:15:31.3826808Z Traceback (most recent call last): 2024-06-26T09:15:31.3827669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2627, in check_accuracy 2024-06-26T09:15:31.3828537Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2024-06-26T09:15:31.3829641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 434, in _fn 2024-06-26T09:15:31.3830419Z return fn(*args, **kwargs) 2024-06-26T09:15:31.3831118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T09:15:31.3831934Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T09:15:31.3832850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 450, in forward_and_backward_pass 2024-06-26T09:15:31.3833687Z cloned_inputs = clone_inputs(inputs) 2024-06-26T09:15:31.3834678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 451, in torch_dynamo_resume_in_forward_and_backward_pass_at_450 2024-06-26T09:15:31.3835647Z self.optimizer_zero_grad(mod) 2024-06-26T09:15:31.3836558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1121, in __call__ 2024-06-26T09:15:31.3837421Z return self._torchdynamo_orig_callable( 2024-06-26T09:15:31.3838353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 948, in __call__ 2024-06-26T09:15:31.3839169Z result = self._inner_convert( 2024-06-26T09:15:31.3840042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 472, in __call__ 2024-06-26T09:15:31.3840844Z return _compile( 2024-06-26T09:15:31.3841673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 85, in wrapper_function 2024-06-26T09:15:31.3842639Z return StrobelightCompileTimeProfiler.profile_compile_time( 2024-06-26T09:15:31.3843838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time 2024-06-26T09:15:31.3844784Z return func(*args, **kwargs) 2024-06-26T09:15:31.3845396Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T09:15:31.3846107Z return func(*args, **kwds) 2024-06-26T09:15:31.3846975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 817, in _compile 2024-06-26T09:15:31.3847922Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-06-26T09:15:31.3848933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T09:15:31.3849728Z r = func(*args, **kwargs) 2024-06-26T09:15:31.3850611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 636, in compile_inner 2024-06-26T09:15:31.3851518Z out_code = transform_code_object(code, transform) 2024-06-26T09:15:31.3852621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1270, in transform_code_object 2024-06-26T09:15:31.3853620Z transformations(instructions, code_options) 2024-06-26T09:15:31.3854539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 178, in _fn 2024-06-26T09:15:31.3855332Z return fn(*args, **kwargs) 2024-06-26T09:15:31.3856258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 590, in transform 2024-06-26T09:15:31.3857170Z bisect(tracer.output.shape_env) 2024-06-26T09:15:31.3858248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/validator.py", line 720, in bisect 2024-06-26T09:15:31.3859404Z last_exception = check_shapeenv_fails(shape_env, shape_env._snapshot_tracked_fakes()) 2024-06-26T09:15:31.3860678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/validator.py", line 702, in check_shapeenv_fails 2024-06-26T09:15:31.3861606Z shape_env.produce_guards( 2024-06-26T09:15:31.3862711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4180, in produce_guards 2024-06-26T09:15:31.3863725Z raise ConstraintViolationError( 2024-06-26T09:15:31.3865102Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-06-26T09:15:31.3866859Z - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0].size()[0]) are valid because L['cloned_inputs'][0].size()[0] was inferred to be a constant (4). 2024-06-26T09:15:31.3867708Z 2024-06-26T09:15:31.3867713Z 2024-06-26T09:15:31.3868015Z You can suppress this exception and fall back to eager by setting: 2024-06-26T09:15:31.3868579Z import torch._dynamo 2024-06-26T09:15:31.3869004Z torch._dynamo.config.suppress_errors = True 2024-06-26T09:15:31.3869364Z 2024-06-26T09:15:31.3869670Z TorchDynamo optimized model failed to run because of following error 2024-06-26T09:15:31.3870264Z fail_to_run 2024-06-26T09:15:31.7371620Z TIMING: entire_frame_compile:0.02147 2024-06-26T09:15:31.7373048Z STATS: call_* op count: 3361 | FakeTensorMode.__torch_dispatch__:150744 | attempt fast:102 | slow both tensors nontrivially broadcast:2 | fast is_contiguous:100 | FakeTensor.__torch_dispatch__:3 | ProxyTorchDispatchMode.__torch_dispatch__:74544 2024-06-26T09:15:31.7374583Z Dynamo produced 0 graphs covering 3361 ops with 3 graph breaks (3 unique) 2024-06-26T09:15:36.3129339Z 2024-06-26T09:15:38.2513146Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:15:38.2513631Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:15:38.2514082Z cuda train hf_DistilBert 2024-06-26T09:16:23.8696693Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:16:23.8698292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:16:23.8699296Z pred = mod(*cloned_inputs) 2024-06-26T09:16:23.8700329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:16:23.8701191Z return forward_call(*args, **kwargs) 2024-06-26T09:16:23.8702268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-06-26T09:16:23.8703520Z dlbrt_output = self.distilbert( 2024-06-26T09:16:23.8704944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:16:23.8705831Z return forward_call(*args, **kwargs) 2024-06-26T09:16:23.8707067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-06-26T09:16:23.8708444Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-06-26T09:16:23.8709833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:16:23.8710677Z return forward_call(*args, **kwargs) 2024-06-26T09:16:23.8711743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-06-26T09:16:23.8712881Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-06-26T09:16:23.8713357Z 2024-06-26T09:16:24.9284368Z W0626 09:16:24.927000 139740415873664 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:16:52.6664625Z pass 2024-06-26T09:16:52.6913570Z TIMING: entire_frame_compile:66.50359 code_gen:14.73631 inductor_compile:35.62977 backend_compile:57.93121 2024-06-26T09:16:52.6916574Z STATS: call_* op count: 553 | FakeTensor.__torch_dispatch__:9366 | FakeTensorMode.__torch_dispatch__:69025 | attempt fast:1695 | fast is_contiguous:1659 | slow no contiguity match:36 | ProxyTorchDispatchMode.__torch_dispatch__:14154 2024-06-26T09:16:52.6918084Z Dynamo produced 3 graphs covering 553 ops with 6 graph breaks (5 unique) 2024-06-26T09:16:59.3605888Z 2024-06-26T09:17:02.3498371Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:17:02.3498964Z loading model: 0it [00:02, ?it/s] 2024-06-26T09:17:02.3499422Z cuda train hf_GPT2 2024-06-26T09:18:17.8823779Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:18:17.8825441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:18:17.8826413Z pred = mod(*cloned_inputs) 2024-06-26T09:18:17.8827429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:18:17.8828284Z return forward_call(*args, **kwargs) 2024-06-26T09:18:17.8829959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-06-26T09:18:17.8844571Z transformer_outputs = self.transformer( 2024-06-26T09:18:17.8846078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:18:17.8847296Z return forward_call(*args, **kwargs) 2024-06-26T09:18:17.8848671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-06-26T09:18:17.8849665Z inputs_embeds = self.wte(input_ids) 2024-06-26T09:18:17.8849972Z 2024-06-26T09:18:19.7589712Z W0626 09:18:19.758000 140208500216448 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:18:57.4610900Z pass 2024-06-26T09:18:57.5363877Z TIMING: entire_frame_compile:105.45005 code_gen:21.11456 inductor_compile:53.27666 backend_compile:90.95861 2024-06-26T09:18:57.5365795Z STATS: call_* op count: 1280 | FakeTensorMode.__torch_dispatch__:103635 | FakeTensor.__torch_dispatch__:13667 | attempt fast:2910 | fast is_contiguous:2886 | ProxyTorchDispatchMode.__torch_dispatch__:21914 | slow no contiguity match:24 2024-06-26T09:18:57.5367484Z Dynamo produced 3 graphs covering 1280 ops with 6 graph breaks (5 unique) 2024-06-26T09:19:06.2497470Z 2024-06-26T09:19:18.6233985Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:19:18.6234527Z loading model: 0it [00:12, ?it/s] 2024-06-26T09:19:18.6235093Z cuda train hf_GPT2_large 2024-06-26T09:19:18.6237735Z pass_due_to_skip 2024-06-26T09:19:19.1192467Z TIMING: 2024-06-26T09:19:19.1192887Z STATS: call_* op count: 0 2024-06-26T09:19:19.1193533Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T09:19:22.4226407Z 2024-06-26T09:19:23.8955105Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:19:23.8955596Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:19:23.8956112Z cuda train hf_Reformer 2024-06-26T09:19:35.1158186Z skipping cudagraphs due to skipping cudagraphs due to cpu device (gcd). Found from : 2024-06-26T09:19:35.1159789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2057, in forward 2024-06-26T09:19:35.1161038Z least_common_mult_chunk_length = _get_least_common_mult_chunk_len(self.config) 2024-06-26T09:19:35.1162510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 91, in _get_least_common_mult_chunk_len 2024-06-26T09:19:35.1163842Z return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length) 2024-06-26T09:19:35.1164378Z 2024-06-26T09:19:37.5190791Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:19:37.5192427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066 2024-06-26T09:19:37.5193553Z embedding_output = self.embeddings( 2024-06-26T09:19:37.5194819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 254, in forward 2024-06-26T09:19:37.5195951Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T09:19:37.5196897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T09:19:37.5197700Z return F.embedding( 2024-06-26T09:19:37.5197969Z 2024-06-26T09:19:43.0795903Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T09:19:45.7281689Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T09:19:47.1136452Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T09:19:48.9962955Z W0626 09:19:48.995000 139933204222592 torch/_logging/_internal.py:1040] [32/0] Profiler function will be ignored 2024-06-26T09:20:16.9956413Z pass 2024-06-26T09:20:17.0075314Z TIMING: entire_frame_compile:36.57149 inductor_compile:22.06473 backend_compile:29.93943 code_gen:14.63768 2024-06-26T09:20:17.0077001Z STATS: call_* op count: 693 | FakeTensorMode.__torch_dispatch__:37625 | ProxyTorchDispatchMode.__torch_dispatch__:8316 | FakeTensor.__torch_dispatch__:4968 | attempt fast:30 | fast is_contiguous:21 | slow no contiguity match:9 2024-06-26T09:20:17.0078478Z Dynamo produced 17 graphs covering 693 ops with 25 graph breaks (9 unique) 2024-06-26T09:20:21.8179594Z 2024-06-26T09:20:25.9120010Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:20:25.9120575Z loading model: 0it [00:04, ?it/s] 2024-06-26T09:20:25.9121031Z cuda train hf_T5_base 2024-06-26T09:20:29.0455641Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2024-06-26T09:20:31.1534779Z ERROR:common: 2024-06-26T09:20:31.1535246Z Traceback (most recent call last): 2024-06-26T09:20:31.1535990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2564, in check_accuracy 2024-06-26T09:20:31.1536809Z correct_rerun_result = self.run_n_iterations( 2024-06-26T09:20:31.1537750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T09:20:31.1538791Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T09:20:31.1539887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T09:20:31.1541009Z self.grad_scaler.scale(loss).backward() 2024-06-26T09:20:31.1542147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T09:20:31.1542909Z torch.autograd.backward( 2024-06-26T09:20:31.1543771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T09:20:31.1544855Z _engine_run_backward( 2024-06-26T09:20:31.1545759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T09:20:31.1546935Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T09:20:31.1550452Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 15991 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T09:20:31.2432810Z eager_2nd_run_OOM 2024-06-26T09:20:31.2463350Z TIMING: 2024-06-26T09:20:31.2463663Z STATS: call_* op count: 0 2024-06-26T09:20:31.2464446Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T09:20:34.5807602Z 2024-06-26T09:20:44.2417631Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:20:44.2418589Z loading model: 0it [00:09, ?it/s] 2024-06-26T09:20:44.2419236Z cuda train hf_T5_large 2024-06-26T09:20:44.2423024Z pass_due_to_skip 2024-06-26T09:20:44.7079188Z TIMING: 2024-06-26T09:20:44.7079551Z STATS: call_* op count: 0 2024-06-26T09:20:44.7080102Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T09:20:48.0064556Z 2024-06-26T09:20:49.0629490Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:20:49.0629980Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:20:49.0630434Z cuda train hf_Whisper 2024-06-26T09:21:20.5603108Z W0626 09:21:20.559000 140164935275136 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:21:41.0692169Z pass 2024-06-26T09:21:41.0797865Z TIMING: entire_frame_compile:45.85374 code_gen:14.81351 inductor_compile:26.92969 backend_compile:40.42654 2024-06-26T09:21:41.0799518Z STATS: call_* op count: 355 | FakeTensor.__torch_dispatch__:6428 | FakeTensorMode.__torch_dispatch__:42271 | attempt fast:975 | fast is_contiguous:919 | slow no contiguity match:56 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2024-06-26T09:21:41.0801046Z Dynamo produced 3 graphs covering 355 ops with 6 graph breaks (5 unique) 2024-06-26T09:21:46.4284274Z 2024-06-26T09:21:46.6975450Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:21:46.6975934Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:21:46.6976390Z cuda train hf_distil_whisper 2024-06-26T09:21:46.6979803Z Traceback (most recent call last): 2024-06-26T09:21:46.6980708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T09:21:46.6981459Z ) = runner.load_model( 2024-06-26T09:21:46.6982402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 320, in load_model 2024-06-26T09:21:46.6983307Z benchmark = benchmark_cls( 2024-06-26T09:21:46.6984406Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T09:21:46.6985320Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T09:21:46.6986426Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__ 2024-06-26T09:21:46.6987443Z raise NotImplementedError("Training is not implemented.") 2024-06-26T09:21:46.6988054Z NotImplementedError: Training is not implemented. 2024-06-26T09:21:46.6988412Z 2024-06-26T09:21:46.6988537Z model_fail_to_load 2024-06-26T09:21:49.8402590Z 2024-06-26T09:21:50.2271984Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:21:50.2272472Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:21:50.2272911Z cuda train lennard_jones 2024-06-26T09:21:54.3711111Z W0626 09:21:54.370000 139709638611584 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:21:58.1166582Z pass 2024-06-26T09:21:58.1167327Z TIMING: entire_frame_compile:6.33713 code_gen:2.25399 inductor_compile:3.92252 backend_compile:5.56279 2024-06-26T09:21:58.1168781Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:870 | FakeTensorMode.__torch_dispatch__:5866 | attempt fast:82 | fast is_contiguous:82 | ProxyTorchDispatchMode.__torch_dispatch__:1221 2024-06-26T09:21:58.1170073Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique) 2024-06-26T09:21:59.2255661Z accuracy pass_rate=72.73% 2024-06-26T09:21:59.2259296Z calls_captured gmean=0.00x mean=447.152x 2024-06-26T09:21:59.2262843Z unique_graphs gmean=0.00x mean=3.061x 2024-06-26T09:21:59.2266303Z graph_breaks gmean=0.00x mean=6.212x 2024-06-26T09:21:59.2270132Z unique_graph_breaks gmean=0.00x mean=4.121x 2024-06-26T09:21:59.2273402Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T09:21:59.2276552Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T09:21:59.2280066Z cudagraph_skips gmean=0.00x mean=0.697x 2024-06-26T09:22:00.0827344Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_torchbench_training.csv 2024-06-26T09:22:00.4040769Z torchrec_dlrm XFAIL 2024-06-26T09:22:00.4044367Z BERT_pytorch PASS 2024-06-26T09:22:00.4049326Z Background_Matting XFAIL 2024-06-26T09:22:00.4054444Z LearningToPaint PASS 2024-06-26T09:22:00.4059597Z Super_SloMo PASS 2024-06-26T09:22:00.4064304Z alexnet PASS 2024-06-26T09:22:00.4069747Z basic_gnn_edgecnn PASS 2024-06-26T09:22:00.4074894Z basic_gnn_gcn PASS 2024-06-26T09:22:00.4080071Z basic_gnn_gin PASS 2024-06-26T09:22:00.4085067Z basic_gnn_sage PASS 2024-06-26T09:22:00.4090285Z dcgan PASS 2024-06-26T09:22:00.4094976Z demucs XFAIL 2024-06-26T09:22:00.4099849Z densenet121 PASS 2024-06-26T09:22:00.4105019Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-06-26T09:22:00.4109969Z dlrm PASS 2024-06-26T09:22:00.4114980Z drq PASS 2024-06-26T09:22:00.4119799Z fastNLP_Bert PASS 2024-06-26T09:22:00.4124788Z functorch_dp_cifar10 PASS 2024-06-26T09:22:00.4129806Z functorch_maml_omniglot PASS 2024-06-26T09:22:00.4134740Z hf_Albert PASS 2024-06-26T09:22:00.4139541Z hf_Bart PASS 2024-06-26T09:22:00.4144220Z hf_Bert PASS 2024-06-26T09:22:00.4149226Z hf_Bert_large PASS 2024-06-26T09:22:00.4153981Z hf_BigBird XFAIL 2024-06-26T09:22:00.4158745Z hf_DistilBert PASS 2024-06-26T09:22:00.4163466Z hf_GPT2 PASS 2024-06-26T09:22:00.4168562Z hf_GPT2_large XFAIL 2024-06-26T09:22:00.4173342Z hf_Reformer PASS 2024-06-26T09:22:00.4177975Z hf_T5_base XFAIL 2024-06-26T09:22:00.4182976Z hf_T5_large XFAIL 2024-06-26T09:22:00.4187770Z hf_Whisper PASS 2024-06-26T09:22:00.4192550Z hf_distil_whisper XFAIL 2024-06-26T09:22:00.4197325Z lennard_jones PASS 2024-06-26T09:22:00.4663402Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_torchbench_training.csv 2024-06-26T09:22:00.7608359Z torchrec_dlrm PASS 2024-06-26T09:22:00.7612957Z BERT_pytorch PASS 2024-06-26T09:22:00.7617208Z Background_Matting PASS 2024-06-26T09:22:00.7620891Z LearningToPaint PASS 2024-06-26T09:22:00.7626837Z Super_SloMo PASS 2024-06-26T09:22:00.7630817Z alexnet PASS 2024-06-26T09:22:00.7635619Z basic_gnn_edgecnn PASS 2024-06-26T09:22:00.7640244Z basic_gnn_gcn PASS 2024-06-26T09:22:00.7644950Z basic_gnn_gin PASS 2024-06-26T09:22:00.7649863Z basic_gnn_sage PASS 2024-06-26T09:22:00.7654401Z dcgan PASS 2024-06-26T09:22:00.7659132Z demucs PASS 2024-06-26T09:22:00.7663733Z densenet121 PASS 2024-06-26T09:22:00.7668771Z detectron2_maskrcnn_r_50_c4 PASS 2024-06-26T09:22:00.7673290Z dlrm PASS 2024-06-26T09:22:00.7678064Z drq PASS 2024-06-26T09:22:00.7682570Z fastNLP_Bert PASS 2024-06-26T09:22:00.7687507Z functorch_dp_cifar10 PASS 2024-06-26T09:22:00.7692345Z functorch_maml_omniglot PASS 2024-06-26T09:22:00.7697250Z hf_Albert PASS 2024-06-26T09:22:00.7701523Z hf_Bart PASS 2024-06-26T09:22:00.7706280Z hf_Bert PASS 2024-06-26T09:22:00.7711129Z hf_Bert_large PASS 2024-06-26T09:22:00.7715724Z hf_BigBird PASS 2024-06-26T09:22:00.7720353Z hf_DistilBert PASS 2024-06-26T09:22:00.7724764Z hf_GPT2 PASS 2024-06-26T09:22:00.7729815Z hf_GPT2_large PASS 2024-06-26T09:22:00.7734284Z hf_Reformer PASS 2024-06-26T09:22:00.7738829Z hf_T5_base PASS 2024-06-26T09:22:00.7743426Z hf_T5_large PASS 2024-06-26T09:22:00.7753845Z hf_Whisper PASS 2024-06-26T09:22:00.7759539Z hf_distil_whisper PASS 2024-06-26T09:22:00.7764581Z lennard_jones PASS 2024-06-26T09:22:00.8249676Z + cleanup_workspace 2024-06-26T09:22:00.8250614Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T09:22:00.8251748Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T09:22:00.8252688Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T09:22:00.8253337Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T09:22:00.8254141Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T09:22:00.8255026Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T09:22:00.8255730Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T09:22:01.9061747Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T09:22:01.9062481Z cat test/**/*_toprint.log || true 2024-06-26T09:22:01.9073021Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:01.9073717Z env: 2024-06-26T09:22:01.9074090Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:01.9074741Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:01.9075779Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:01.9076698Z ##[endgroup] 2024-06-26T09:22:01.9143500Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T09:22:01.9171902Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T09:22:01.9172363Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T09:22:01.9180031Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:01.9180539Z env: 2024-06-26T09:22:01.9180822Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:01.9181285Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:01.9182035Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:01.9182704Z MONITOR_SCRIPT_PID: 22581 2024-06-26T09:22:01.9183059Z ##[endgroup] 2024-06-26T09:22:01.9372564Z Prepare all required actions 2024-06-26T09:22:01.9373005Z Getting action download info 2024-06-26T09:22:02.0888546Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T09:22:02.2558333Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T09:22:02.2558802Z with: 2024-06-26T09:22:02.2559385Z file-suffix: test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970 2024-06-26T09:22:02.2560091Z s3-bucket: gha-artifacts 2024-06-26T09:22:02.2560433Z env: 2024-06-26T09:22:02.2560700Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:02.2561158Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:02.2561912Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:02.2562571Z ##[endgroup] 2024-06-26T09:22:02.2587352Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T09:22:02.2588014Z # Remove any previous test jsons if they exist 2024-06-26T09:22:02.2588540Z rm -f test-jsons-*.zip 2024-06-26T09:22:02.2589192Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T09:22:02.2597179Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:02.2597687Z env: 2024-06-26T09:22:02.2597978Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:02.2598441Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:02.2599204Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:02.2600153Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970 2024-06-26T09:22:02.2600834Z ##[endgroup] 2024-06-26T09:22:02.2774182Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T09:22:02.2802028Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T09:22:02.2802723Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T09:22:02.2808465Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T09:22:02.2835311Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T09:22:02.2835966Z # Remove any previous test reports if they exist 2024-06-26T09:22:02.2836499Z rm -f test-reports-*.zip 2024-06-26T09:22:02.2837099Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T09:22:02.2844760Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:02.2845258Z env: 2024-06-26T09:22:02.2845628Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:02.2846091Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:02.2846896Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:02.2847833Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970 2024-06-26T09:22:02.2848517Z ##[endgroup] 2024-06-26T09:22:02.3010670Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-06-26T09:22:02.3014900Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-06-26T09:22:02.3017051Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-26T09:22:02.3018087Z adding: test/test-reports/training_torchbench.csv (deflated 63%) 2024-06-26T09:22:02.3022811Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-26T09:22:02.3024121Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 81%) 2024-06-26T09:22:02.3049706Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T09:22:02.3050338Z # Remove any previous usage logs if they exist 2024-06-26T09:22:02.3050840Z rm -f logs-*.zip 2024-06-26T09:22:02.3051514Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T09:22:02.3052411Z # so check to see if the file exists first 2024-06-26T09:22:02.3052916Z if [ -f 'usage_log.txt' ]; then 2024-06-26T09:22:02.3053456Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T09:22:02.3053946Z fi 2024-06-26T09:22:02.3054310Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T09:22:02.3054882Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T09:22:02.3055377Z fi 2024-06-26T09:22:02.3062620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:02.3063117Z env: 2024-06-26T09:22:02.3063396Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:02.3064045Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:02.3064796Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:02.3065737Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970 2024-06-26T09:22:02.3066410Z ##[endgroup] 2024-06-26T09:22:02.3159428Z adding: usage_log.txt (deflated 92%) 2024-06-26T09:22:02.3225481Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T09:22:02.3226323Z # Remove any previous debugging artifacts if they exist 2024-06-26T09:22:02.3226883Z rm -f debug-*.zip 2024-06-26T09:22:02.3227276Z if [ -d 'test/debug' ]; then 2024-06-26T09:22:02.3227789Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T09:22:02.3228283Z fi 2024-06-26T09:22:02.3235117Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:02.3235624Z env: 2024-06-26T09:22:02.3235906Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:02.3236378Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:02.3237169Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:02.3238133Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970 2024-06-26T09:22:02.3238823Z ##[endgroup] 2024-06-26T09:22:02.3312463Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:22:02.3312912Z with: 2024-06-26T09:22:02.3313199Z s3-bucket: gha-artifacts 2024-06-26T09:22:02.3313639Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:22:02.3314111Z retention-days: 14 2024-06-26T09:22:02.3314451Z if-no-files-found: warn 2024-06-26T09:22:02.3314817Z path: test-jsons-*.zip 2024-06-26T09:22:02.3315165Z name: artifact 2024-06-26T09:22:02.3315472Z region: us-east-1 2024-06-26T09:22:02.3315782Z env: 2024-06-26T09:22:02.3316066Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:02.3316531Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:02.3317282Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:02.3317933Z ##[endgroup] 2024-06-26T09:22:02.6475529Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:22:02.6476226Z With the provided path, there will be 1 file uploaded 2024-06-26T09:22:02.6476874Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:22:02.6502505Z Starting upload of test-jsons-test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970.zip 2024-06-26T09:22:02.7664814Z Finished upload of test-jsons-test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970.zip 2024-06-26T09:22:02.7806940Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:22:02.7807388Z with: 2024-06-26T09:22:02.7807666Z s3-bucket: gha-artifacts 2024-06-26T09:22:02.7808095Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:22:02.7808568Z retention-days: 14 2024-06-26T09:22:02.7808897Z if-no-files-found: error 2024-06-26T09:22:02.7809265Z path: test-reports-*.zip 2024-06-26T09:22:02.7809621Z name: artifact 2024-06-26T09:22:02.7809929Z region: us-east-1 2024-06-26T09:22:02.7810225Z env: 2024-06-26T09:22:02.7810502Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:02.7811108Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:02.7811855Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:02.7812512Z ##[endgroup] 2024-06-26T09:22:03.1019656Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:22:03.1020567Z With the provided path, there will be 1 file uploaded 2024-06-26T09:22:03.1021372Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:22:03.1048906Z Starting upload of test-reports-test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970.zip 2024-06-26T09:22:03.5015698Z Finished upload of test-reports-test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970.zip 2024-06-26T09:22:03.5155464Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:22:03.5155916Z with: 2024-06-26T09:22:03.5156199Z s3-bucket: gha-artifacts 2024-06-26T09:22:03.5156629Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:22:03.5157107Z retention-days: 14 2024-06-26T09:22:03.5157463Z if-no-files-found: ignore 2024-06-26T09:22:03.5157834Z path: logs-*.zip 2024-06-26T09:22:03.5158282Z name: artifact 2024-06-26T09:22:03.5158591Z region: us-east-1 2024-06-26T09:22:03.5158901Z env: 2024-06-26T09:22:03.5159184Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:03.5159650Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:03.5160403Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:03.5161056Z ##[endgroup] 2024-06-26T09:22:03.8257131Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:22:03.8257832Z With the provided path, there will be 1 file uploaded 2024-06-26T09:22:03.8258489Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:22:03.8284225Z Starting upload of logs-test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970.zip 2024-06-26T09:22:03.9808143Z Finished upload of logs-test-dynamic_inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911970.zip 2024-06-26T09:22:03.9954185Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:22:03.9954648Z with: 2024-06-26T09:22:03.9954951Z s3-bucket: gha-artifacts 2024-06-26T09:22:03.9955394Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:22:03.9955874Z retention-days: 14 2024-06-26T09:22:03.9956207Z if-no-files-found: ignore 2024-06-26T09:22:03.9956575Z path: debug-*.zip 2024-06-26T09:22:03.9956894Z name: artifact 2024-06-26T09:22:03.9957198Z region: us-east-1 2024-06-26T09:22:03.9957502Z env: 2024-06-26T09:22:03.9957787Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:03.9958257Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:03.9959005Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:03.9959651Z ##[endgroup] 2024-06-26T09:22:04.3008405Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T09:22:04.3150943Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T09:22:04.3151438Z # shellcheck disable=SC2156 2024-06-26T09:22:04.3152254Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T09:22:04.3160176Z shell: /usr/bin/bash -e {0} 2024-06-26T09:22:04.3160531Z env: 2024-06-26T09:22:04.3160816Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:04.3161280Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:04.3162023Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:04.3162677Z ##[endgroup] 2024-06-26T09:22:04.4846583Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T09:22:04.4847162Z with: 2024-06-26T09:22:04.4847431Z env: 2024-06-26T09:22:04.4847722Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:04.4848183Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:04.4848950Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:04.4849603Z ##[endgroup] 2024-06-26T09:22:04.4867384Z ##[group]Run set -eou pipefail 2024-06-26T09:22:04.4867805Z set -eou pipefail 2024-06-26T09:22:04.4868170Z  2024-06-26T09:22:04.4868698Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T09:22:04.4869361Z for _ in $(seq 1440); do 2024-06-26T09:22:04.4869872Z  # Break if no ssh session exists anymore 2024-06-26T09:22:04.4870403Z  if [ "$(who)" = "" ]; then 2024-06-26T09:22:04.4870845Z  break 2024-06-26T09:22:04.4871206Z  fi 2024-06-26T09:22:04.4871525Z  echo "." 2024-06-26T09:22:04.4871879Z  sleep 5 2024-06-26T09:22:04.4872223Z done 2024-06-26T09:22:04.4880077Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:04.4880595Z env: 2024-06-26T09:22:04.4880908Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:04.4881393Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:04.4882147Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:04.4882935Z ##[endgroup] 2024-06-26T09:22:04.4902737Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T09:22:04.4939322Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T09:22:04.4940092Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T09:22:04.4940691Z # shellcheck disable=SC2046 2024-06-26T09:22:04.4941162Z docker stop $(docker ps -q) || true 2024-06-26T09:22:04.4941655Z # Prune all of the docker images 2024-06-26T09:22:04.4942115Z docker system prune -af 2024-06-26T09:22:04.4949210Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:04.4949720Z env: 2024-06-26T09:22:04.4950004Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:22:04.4950473Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:22:04.4951231Z DOCKER_CONTAINER_ID: 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:04.4951892Z ##[endgroup] 2024-06-26T09:22:04.9218907Z 41defb39909c 2024-06-26T09:22:07.2177068Z Deleted Containers: 2024-06-26T09:22:07.2177808Z 41defb39909c55e466d882e7cc1bfbe0d193589ed5dbd8fff1fde319f1b6cc76 2024-06-26T09:22:07.2178363Z 2024-06-26T09:22:11.2557832Z Deleted Images: 2024-06-26T09:22:11.2559789Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T09:22:11.2562233Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T09:22:11.2564243Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T09:22:11.2565155Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T09:22:11.2566110Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T09:22:11.2566958Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T09:22:11.2567816Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T09:22:11.2568658Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T09:22:11.2569490Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T09:22:11.2570539Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T09:22:11.2571374Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T09:22:11.2572188Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T09:22:11.2573014Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T09:22:11.2573833Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T09:22:11.2574661Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T09:22:11.2575496Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T09:22:11.2576341Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T09:22:11.2577164Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T09:22:11.2577999Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T09:22:11.2578826Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T09:22:11.2579718Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T09:22:11.2580553Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T09:22:11.2581368Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T09:22:11.2582321Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T09:22:11.2583143Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T09:22:11.2584229Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T09:22:11.2585267Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T09:22:11.2586114Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T09:22:11.2586963Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T09:22:11.2587816Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T09:22:11.2588644Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T09:22:11.2606224Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T09:22:11.2607139Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T09:22:11.2607974Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T09:22:11.2608820Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T09:22:11.2609682Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T09:22:11.2610502Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T09:22:11.2611332Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T09:22:11.2612160Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T09:22:11.2612986Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T09:22:11.2613798Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T09:22:11.2614627Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T09:22:11.2615467Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T09:22:11.2616300Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T09:22:11.2617100Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T09:22:11.2617937Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T09:22:11.2618954Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T09:22:11.2619818Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T09:22:11.2620636Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T09:22:11.2621467Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T09:22:11.2622300Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T09:22:11.2623142Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T09:22:11.2624265Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T09:22:11.2625109Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T09:22:11.2625947Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T09:22:11.2626794Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T09:22:11.2627634Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T09:22:11.2628458Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T09:22:11.2629288Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T09:22:11.2630133Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T09:22:11.2631055Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T09:22:11.2631884Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T09:22:11.2632736Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T09:22:11.2633746Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T09:22:11.2634591Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T09:22:11.2635423Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T09:22:11.2636249Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T09:22:11.2637066Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T09:22:11.2637903Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T09:22:11.2638720Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T09:22:11.2639558Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T09:22:11.2640396Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T09:22:11.2641221Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T09:22:11.2642054Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T09:22:11.2642894Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T09:22:11.2643887Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T09:22:11.2644813Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T09:22:11.2645806Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T09:22:11.2646788Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T09:22:11.2647754Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T09:22:11.2648580Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T09:22:11.2649077Z 2024-06-26T09:22:11.2649240Z Total reclaimed space: 40.92GB 2024-06-26T09:22:11.2723924Z Post job cleanup. 2024-06-26T09:22:11.2772866Z Post job cleanup. 2024-06-26T09:22:11.3599482Z [command]/usr/bin/git version 2024-06-26T09:22:11.3643927Z git version 2.40.1 2024-06-26T09:22:11.3683759Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5c06d25c-10c3-4460-b3f1-958ea7f079f4' before making global git config changes 2024-06-26T09:22:11.3685156Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T09:22:11.3689637Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T09:22:11.3715771Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T09:22:11.3738049Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T09:22:11.3956133Z Entering 'android/libs/fbjni' 2024-06-26T09:22:11.3991853Z Entering 'third_party/FP16' 2024-06-26T09:22:11.4029359Z Entering 'third_party/FXdiv' 2024-06-26T09:22:11.4066949Z Entering 'third_party/NNPACK' 2024-06-26T09:22:11.4102326Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T09:22:11.4138697Z Entering 'third_party/XNNPACK' 2024-06-26T09:22:11.4187069Z Entering 'third_party/benchmark' 2024-06-26T09:22:11.4224247Z Entering 'third_party/cpp-httplib' 2024-06-26T09:22:11.4257619Z Entering 'third_party/cpuinfo' 2024-06-26T09:22:11.4293541Z Entering 'third_party/cudnn_frontend' 2024-06-26T09:22:11.4329427Z Entering 'third_party/cutlass' 2024-06-26T09:22:11.4370242Z Entering 'third_party/eigen' 2024-06-26T09:22:11.4408737Z Entering 'third_party/fbgemm' 2024-06-26T09:22:11.4445331Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T09:22:11.4483573Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T09:22:11.4521221Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T09:22:11.4560093Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T09:22:11.4595045Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T09:22:11.4631458Z Entering 'third_party/flatbuffers' 2024-06-26T09:22:11.4670259Z Entering 'third_party/fmt' 2024-06-26T09:22:11.4707175Z Entering 'third_party/foxi' 2024-06-26T09:22:11.4743250Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T09:22:11.4778863Z Entering 'third_party/gloo' 2024-06-26T09:22:11.4816517Z Entering 'third_party/googletest' 2024-06-26T09:22:11.4852232Z Entering 'third_party/ideep' 2024-06-26T09:22:11.4888642Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T09:22:11.4930573Z Entering 'third_party/ittapi' 2024-06-26T09:22:11.4963557Z Entering 'third_party/kineto' 2024-06-26T09:22:11.4999536Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T09:22:11.5034052Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T09:22:11.5068184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T09:22:11.5102741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T09:22:11.5136661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T09:22:11.5171033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T09:22:11.5206314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T09:22:11.5241217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T09:22:11.5273509Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T09:22:11.5309133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T09:22:11.5343688Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T09:22:11.5378496Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T09:22:11.5414602Z Entering 'third_party/mimalloc' 2024-06-26T09:22:11.5451414Z Entering 'third_party/nccl/nccl' 2024-06-26T09:22:11.5487521Z Entering 'third_party/nlohmann' 2024-06-26T09:22:11.5521571Z Entering 'third_party/onnx' 2024-06-26T09:22:11.5565674Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T09:22:11.5600970Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T09:22:11.5636956Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T09:22:11.5672065Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T09:22:11.5706302Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T09:22:11.5740963Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T09:22:11.5775908Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T09:22:11.5811288Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T09:22:11.5845821Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T09:22:11.5879085Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T09:22:11.5912738Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T09:22:11.5948966Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T09:22:11.5985878Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T09:22:11.6034786Z Entering 'third_party/pocketfft' 2024-06-26T09:22:11.6068262Z Entering 'third_party/protobuf' 2024-06-26T09:22:11.6105557Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T09:22:11.6137115Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T09:22:11.6174048Z Entering 'third_party/psimd' 2024-06-26T09:22:11.6210451Z Entering 'third_party/pthreadpool' 2024-06-26T09:22:11.6243641Z Entering 'third_party/pybind11' 2024-06-26T09:22:11.6279580Z Entering 'third_party/python-peachpy' 2024-06-26T09:22:11.6315793Z Entering 'third_party/sleef' 2024-06-26T09:22:11.6349111Z Entering 'third_party/tensorpipe' 2024-06-26T09:22:11.6384673Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T09:22:11.6419157Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T09:22:11.6453665Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T09:22:11.6489031Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T09:22:11.6522488Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T09:22:11.6573094Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T09:22:11.6591412Z http.https://github.com/.extraheader 2024-06-26T09:22:11.6599070Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T09:22:11.6628048Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T09:22:11.6834456Z Entering 'android/libs/fbjni' 2024-06-26T09:22:11.6855361Z http.https://github.com/.extraheader 2024-06-26T09:22:11.6878519Z Entering 'third_party/FP16' 2024-06-26T09:22:11.6900031Z http.https://github.com/.extraheader 2024-06-26T09:22:11.6924828Z Entering 'third_party/FXdiv' 2024-06-26T09:22:11.6945976Z http.https://github.com/.extraheader 2024-06-26T09:22:11.6970798Z Entering 'third_party/NNPACK' 2024-06-26T09:22:11.6993538Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7017901Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T09:22:11.7039944Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7065294Z Entering 'third_party/XNNPACK' 2024-06-26T09:22:11.7086341Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7118916Z Entering 'third_party/benchmark' 2024-06-26T09:22:11.7141730Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7164898Z Entering 'third_party/cpp-httplib' 2024-06-26T09:22:11.7187602Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7211421Z Entering 'third_party/cpuinfo' 2024-06-26T09:22:11.7233995Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7257044Z Entering 'third_party/cudnn_frontend' 2024-06-26T09:22:11.7279426Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7303023Z Entering 'third_party/cutlass' 2024-06-26T09:22:11.7326008Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7353944Z Entering 'third_party/eigen' 2024-06-26T09:22:11.7376112Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7401123Z Entering 'third_party/fbgemm' 2024-06-26T09:22:11.7422329Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7444559Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T09:22:11.7466618Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7491375Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T09:22:11.7512263Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7536135Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T09:22:11.7557902Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7584500Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T09:22:11.7606469Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7628690Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T09:22:11.7649711Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7673409Z Entering 'third_party/flatbuffers' 2024-06-26T09:22:11.7696186Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7722867Z Entering 'third_party/fmt' 2024-06-26T09:22:11.7746290Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7771052Z Entering 'third_party/foxi' 2024-06-26T09:22:11.7793475Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7815654Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T09:22:11.7838094Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7861445Z Entering 'third_party/gloo' 2024-06-26T09:22:11.7884740Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7908939Z Entering 'third_party/googletest' 2024-06-26T09:22:11.7930304Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7953394Z Entering 'third_party/ideep' 2024-06-26T09:22:11.7974831Z http.https://github.com/.extraheader 2024-06-26T09:22:11.7996064Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T09:22:11.8017221Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8045121Z Entering 'third_party/ittapi' 2024-06-26T09:22:11.8066409Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8090009Z Entering 'third_party/kineto' 2024-06-26T09:22:11.8112725Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8136078Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T09:22:11.8157768Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8180772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T09:22:11.8203919Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8229258Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T09:22:11.8251082Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8275337Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T09:22:11.8297378Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8320895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T09:22:11.8342015Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8363194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T09:22:11.8384594Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8408606Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T09:22:11.8429781Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8452936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T09:22:11.8475046Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8499922Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T09:22:11.8521265Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8545370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T09:22:11.8566839Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8590538Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T09:22:11.8610600Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8632696Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T09:22:11.8653725Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8678860Z Entering 'third_party/mimalloc' 2024-06-26T09:22:11.8700841Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8724640Z Entering 'third_party/nccl/nccl' 2024-06-26T09:22:11.8747340Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8769747Z Entering 'third_party/nlohmann' 2024-06-26T09:22:11.8792637Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8817570Z Entering 'third_party/onnx' 2024-06-26T09:22:11.8839624Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8877790Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T09:22:11.8899445Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8922163Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T09:22:11.8944783Z http.https://github.com/.extraheader 2024-06-26T09:22:11.8968720Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T09:22:11.8990772Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9014478Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T09:22:11.9036645Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9060468Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T09:22:11.9081692Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9104984Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T09:22:11.9126852Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9148569Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T09:22:11.9169392Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9194146Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T09:22:11.9215219Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9240157Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T09:22:11.9262921Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9286524Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T09:22:11.9307472Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9332211Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T09:22:11.9353806Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9377754Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T09:22:11.9399567Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9423732Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T09:22:11.9445248Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9483676Z Entering 'third_party/pocketfft' 2024-06-26T09:22:11.9506146Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9529744Z Entering 'third_party/protobuf' 2024-06-26T09:22:11.9551152Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9576271Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T09:22:11.9597733Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9621754Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T09:22:11.9643475Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9669142Z Entering 'third_party/psimd' 2024-06-26T09:22:11.9690447Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9712182Z Entering 'third_party/pthreadpool' 2024-06-26T09:22:11.9733217Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9757406Z Entering 'third_party/pybind11' 2024-06-26T09:22:11.9778471Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9801997Z Entering 'third_party/python-peachpy' 2024-06-26T09:22:11.9823111Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9845422Z Entering 'third_party/sleef' 2024-06-26T09:22:11.9867267Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9890351Z Entering 'third_party/tensorpipe' 2024-06-26T09:22:11.9911911Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9936071Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T09:22:11.9957886Z http.https://github.com/.extraheader 2024-06-26T09:22:11.9980780Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T09:22:12.0002437Z http.https://github.com/.extraheader 2024-06-26T09:22:12.0026264Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T09:22:12.0047717Z http.https://github.com/.extraheader 2024-06-26T09:22:12.0070476Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T09:22:12.0091569Z http.https://github.com/.extraheader 2024-06-26T09:22:12.0111705Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T09:22:12.0132932Z http.https://github.com/.extraheader 2024-06-26T09:22:12.0233121Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T09:22:12.0250530Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T09:22:12.0257126Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:22:12.0257645Z ##[endgroup] 2024-06-26T09:22:13.2011151Z Cleaning up orphan processes