2024-06-26T07:03:52.6054969Z Current runner version: '2.317.0' 2024-06-26T07:03:52.6061204Z Runner name: 'i-08e7836a94a47ea58' 2024-06-26T07:03:52.6061998Z Runner group name: 'Default' 2024-06-26T07:03:52.6062921Z Machine name: 'ip-10-0-2-254' 2024-06-26T07:03:52.6068019Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:03:52.6070049Z Actions: read 2024-06-26T07:03:52.6070578Z Attestations: read 2024-06-26T07:03:52.6071126Z Checks: read 2024-06-26T07:03:52.6071669Z Contents: read 2024-06-26T07:03:52.6072200Z Deployments: read 2024-06-26T07:03:52.6072718Z Discussions: read 2024-06-26T07:03:52.6073461Z Issues: read 2024-06-26T07:03:52.6073974Z Metadata: read 2024-06-26T07:03:52.6074516Z Packages: read 2024-06-26T07:03:52.6075025Z Pages: read 2024-06-26T07:03:52.6075535Z PullRequests: read 2024-06-26T07:03:52.6076164Z RepositoryProjects: read 2024-06-26T07:03:52.6076771Z SecurityEvents: read 2024-06-26T07:03:52.6077332Z Statuses: read 2024-06-26T07:03:52.6077872Z ##[endgroup] 2024-06-26T07:03:52.6081154Z Secret source: Actions 2024-06-26T07:03:52.6082123Z Prepare workflow directory 2024-06-26T07:03:52.9794158Z Prepare all required actions 2024-06-26T07:03:52.9992222Z Getting action download info 2024-06-26T07:03:53.1749195Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:03:53.6241238Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:03:56.4742590Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:03:56.5922856Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:03:56.9231749Z Getting action download info 2024-06-26T07:03:57.0166951Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:03:57.2681929Z Getting action download info 2024-06-26T07:03:57.3724618Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:03:57.5849923Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:03:57.5852239Z ##[group] Inputs 2024-06-26T07:03:57.5852791Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:03:57.5859851Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:03:57.5868010Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:57.5869254Z sync-tag: 2024-06-26T07:03:57.5870292Z timeout-minutes: 240 2024-06-26T07:03:57.5870708Z use-gha: 2024-06-26T07:03:57.5871079Z dashboard-tag: 2024-06-26T07:03:57.5871486Z s3-bucket: gha-artifacts 2024-06-26T07:03:57.5871912Z aws-role-to-assume: 2024-06-26T07:03:57.5872328Z ##[endgroup] 2024-06-26T07:03:57.5873461Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:03:57.6464009Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:03:57.6607877Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:03:57.6618783Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:57.6619394Z ##[endgroup] 2024-06-26T07:03:58.4307914Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:03:58.4308464Z with: 2024-06-26T07:03:58.4309192Z github-secret: *** 2024-06-26T07:03:58.4310210Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:03:58.4311221Z activate-with-label: false 2024-06-26T07:03:58.4311593Z label: with-ssh 2024-06-26T07:03:58.4311921Z remove-existing-keys: true 2024-06-26T07:03:58.4312286Z fail-silently: true 2024-06-26T07:03:58.4312609Z env: 2024-06-26T07:03:58.4312890Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:58.4313238Z ##[endgroup] 2024-06-26T07:03:58.5176716Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:03:58.5177742Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:03:58.8049563Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:03:58.8810690Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:03:58.8811411Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:03:58.9570323Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:03:58.9680460Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:03:58.9681106Z with: 2024-06-26T07:03:58.9681452Z submodules: recursive 2024-06-26T07:03:58.9681848Z fetch-depth: 0 2024-06-26T07:03:58.9682308Z env: 2024-06-26T07:03:58.9682695Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:58.9683203Z ##[endgroup] 2024-06-26T07:03:58.9877978Z ##[group]Run retry () { 2024-06-26T07:03:58.9878549Z retry () { 2024-06-26T07:03:58.9879186Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:03:58.9879862Z } 2024-06-26T07:03:58.9880345Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:03:58.9880882Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:03:58.9881539Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:03:58.9882213Z else 2024-06-26T07:03:58.9882682Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:03:58.9883242Z fi 2024-06-26T07:03:58.9883730Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:03:58.9892040Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:58.9892664Z env: 2024-06-26T07:03:58.9893060Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:58.9893511Z NO_SUDO: 2024-06-26T07:03:58.9893909Z ##[endgroup] 2024-06-26T07:03:58.9947317Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:01.3220615Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:04:01.3221099Z with: 2024-06-26T07:04:01.3221415Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:01.3221851Z fetch-depth: 0 2024-06-26T07:04:01.3222167Z submodules: recursive 2024-06-26T07:04:01.3222506Z quiet-checkout: true 2024-06-26T07:04:01.3222980Z repository: pytorch/pytorch 2024-06-26T07:04:01.3223615Z token: *** 2024-06-26T07:04:01.3223910Z ssh-strict: true 2024-06-26T07:04:01.3224238Z persist-credentials: true 2024-06-26T07:04:01.3224585Z clean: true 2024-06-26T07:04:01.3224917Z sparse-checkout-cone-mode: true 2024-06-26T07:04:01.3225306Z lfs: false 2024-06-26T07:04:01.3225599Z set-safe-directory: true 2024-06-26T07:04:01.3225938Z env: 2024-06-26T07:04:01.3226213Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:01.3226556Z ##[endgroup] 2024-06-26T07:04:01.4140954Z Syncing repository: pytorch/pytorch 2024-06-26T07:04:01.4142488Z ##[group]Getting Git version info 2024-06-26T07:04:01.4143219Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:01.4144212Z [command]/usr/bin/git version 2024-06-26T07:04:01.4144583Z git version 2.40.1 2024-06-26T07:04:01.4145887Z ##[endgroup] 2024-06-26T07:04:01.4156886Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f309a562-8fd7-48b7-9e84-9092e9e72439' before making global git config changes 2024-06-26T07:04:01.4158115Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:04:01.4161146Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:01.4188693Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:04:01.4192394Z ##[group]Initializing the repository 2024-06-26T07:04:01.4195182Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:04:01.4218872Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:04:01.4219874Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:04:01.4220752Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:04:01.4221311Z hint: 2024-06-26T07:04:01.4221864Z hint: git config --global init.defaultBranch 2024-06-26T07:04:01.4222340Z hint: 2024-06-26T07:04:01.4222845Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:04:01.4223845Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:04:01.4224596Z hint: 2024-06-26T07:04:01.4224907Z hint: git branch -m 2024-06-26T07:04:01.4225639Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:04:01.4227446Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:04:01.4246768Z ##[endgroup] 2024-06-26T07:04:01.4247341Z ##[group]Disabling automatic garbage collection 2024-06-26T07:04:01.4249407Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:04:01.4269072Z ##[endgroup] 2024-06-26T07:04:01.4269579Z ##[group]Setting up auth 2024-06-26T07:04:01.4274911Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:04:01.4294528Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:04:01.4487516Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:04:01.4509970Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:04:01.4709903Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:04:01.4751167Z ##[endgroup] 2024-06-26T07:04:01.4751949Z ##[group]Fetching the repository 2024-06-26T07:04:01.4758087Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:04:03.8557922Z remote: Enumerating objects: 987619 2024-06-26T07:04:03.8558748Z remote: Enumerating objects: 990209, done. 2024-06-26T07:04:03.8561535Z remote: Counting objects: 0% (1/2590) 2024-06-26T07:04:03.8562124Z remote: Counting objects: 1% (26/2590) 2024-06-26T07:04:03.8562647Z remote: Counting objects: 2% (52/2590) 2024-06-26T07:04:03.8563305Z remote: Counting objects: 3% (78/2590) 2024-06-26T07:04:03.8564095Z remote: Counting objects: 4% (104/2590) 2024-06-26T07:04:03.8566037Z remote: Counting objects: 5% (130/2590) 2024-06-26T07:04:03.8566608Z remote: Counting objects: 6% (156/2590) 2024-06-26T07:04:03.8567123Z remote: Counting objects: 7% (182/2590) 2024-06-26T07:04:03.8567866Z remote: Counting objects: 8% (208/2590) 2024-06-26T07:04:03.8569193Z remote: Counting objects: 9% (234/2590) 2024-06-26T07:04:03.8569703Z remote: Counting objects: 10% (259/2590) 2024-06-26T07:04:03.8570347Z remote: Counting objects: 11% (285/2590) 2024-06-26T07:04:03.8570984Z remote: Counting objects: 12% (311/2590) 2024-06-26T07:04:03.8571808Z remote: Counting objects: 13% (337/2590) 2024-06-26T07:04:03.8572510Z remote: Counting objects: 14% (363/2590) 2024-06-26T07:04:03.8573029Z remote: Counting objects: 15% (389/2590) 2024-06-26T07:04:03.8573866Z remote: Counting objects: 16% (415/2590) 2024-06-26T07:04:03.8574674Z remote: Counting objects: 17% (441/2590) 2024-06-26T07:04:03.8575554Z remote: Counting objects: 18% (467/2590) 2024-06-26T07:04:03.8577183Z remote: Counting objects: 19% (493/2590) 2024-06-26T07:04:03.8577783Z remote: Counting objects: 20% (518/2590) 2024-06-26T07:04:03.8578364Z remote: Counting objects: 21% (544/2590) 2024-06-26T07:04:03.8580027Z remote: Counting objects: 22% (570/2590) 2024-06-26T07:04:03.8581774Z remote: Counting objects: 23% (596/2590) 2024-06-26T07:04:03.8582364Z remote: Counting objects: 24% (622/2590) 2024-06-26T07:04:03.8584526Z remote: Counting objects: 25% (648/2590) 2024-06-26T07:04:03.8589472Z remote: Counting objects: 26% (674/2590) 2024-06-26T07:04:03.8590130Z remote: Counting objects: 27% (700/2590) 2024-06-26T07:04:03.8590975Z remote: Counting objects: 28% (726/2590) 2024-06-26T07:04:03.8591639Z remote: Counting objects: 29% (752/2590) 2024-06-26T07:04:03.8592227Z remote: Counting objects: 30% (777/2590) 2024-06-26T07:04:03.8593002Z remote: Counting objects: 31% (803/2590) 2024-06-26T07:04:03.8593580Z remote: Counting objects: 32% (829/2590) 2024-06-26T07:04:03.8594199Z remote: Counting objects: 33% (855/2590) 2024-06-26T07:04:03.8594776Z remote: Counting objects: 34% (881/2590) 2024-06-26T07:04:03.8595384Z remote: Counting objects: 35% (907/2590) 2024-06-26T07:04:03.8595956Z remote: Counting objects: 36% (933/2590) 2024-06-26T07:04:03.8596532Z remote: Counting objects: 37% (959/2590) 2024-06-26T07:04:03.8597118Z remote: Counting objects: 38% (985/2590) 2024-06-26T07:04:03.8597703Z remote: Counting objects: 39% (1011/2590) 2024-06-26T07:04:03.8598302Z remote: Counting objects: 40% (1036/2590) 2024-06-26T07:04:03.8598903Z remote: Counting objects: 41% (1062/2590) 2024-06-26T07:04:03.8599502Z remote: Counting objects: 42% (1088/2590) 2024-06-26T07:04:03.8600097Z remote: Counting objects: 43% (1114/2590) 2024-06-26T07:04:03.8600680Z remote: Counting objects: 44% (1140/2590) 2024-06-26T07:04:03.8602595Z remote: Counting objects: 45% (1166/2590) 2024-06-26T07:04:03.8611237Z remote: Counting objects: 46% (1192/2590) 2024-06-26T07:04:03.8611853Z remote: Counting objects: 47% (1218/2590) 2024-06-26T07:04:03.8612444Z remote: Counting objects: 48% (1244/2590) 2024-06-26T07:04:03.8613042Z remote: Counting objects: 49% (1270/2590) 2024-06-26T07:04:03.8613638Z remote: Counting objects: 50% (1295/2590) 2024-06-26T07:04:03.8616756Z remote: Counting objects: 51% (1321/2590) 2024-06-26T07:04:03.8617413Z remote: Counting objects: 52% (1347/2590) 2024-06-26T07:04:03.8618111Z remote: Counting objects: 53% (1373/2590) 2024-06-26T07:04:03.8618765Z remote: Counting objects: 54% (1399/2590) 2024-06-26T07:04:03.8619412Z remote: Counting objects: 55% (1425/2590) 2024-06-26T07:04:03.8620046Z remote: Counting objects: 56% (1451/2590) 2024-06-26T07:04:03.8620653Z remote: Counting objects: 57% (1477/2590) 2024-06-26T07:04:03.8621351Z remote: Counting objects: 58% (1503/2590) 2024-06-26T07:04:03.8621998Z remote: Counting objects: 59% (1529/2590) 2024-06-26T07:04:03.8622598Z remote: Counting objects: 60% (1554/2590) 2024-06-26T07:04:03.8623553Z remote: Counting objects: 61% (1580/2590) 2024-06-26T07:04:03.8624352Z remote: Counting objects: 62% (1606/2590) 2024-06-26T07:04:03.8625042Z remote: Counting objects: 63% (1632/2590) 2024-06-26T07:04:03.8625802Z remote: Counting objects: 64% (1658/2590) 2024-06-26T07:04:03.8626494Z remote: Counting objects: 65% (1684/2590) 2024-06-26T07:04:03.8627087Z remote: Counting objects: 66% (1710/2590) 2024-06-26T07:04:03.8627725Z remote: Counting objects: 67% (1736/2590) 2024-06-26T07:04:03.8628311Z remote: Counting objects: 68% (1762/2590) 2024-06-26T07:04:03.8628898Z remote: Counting objects: 69% (1788/2590) 2024-06-26T07:04:03.8629472Z remote: Counting objects: 70% (1813/2590) 2024-06-26T07:04:03.8630058Z remote: Counting objects: 71% (1839/2590) 2024-06-26T07:04:03.8630646Z remote: Counting objects: 72% (1865/2590) 2024-06-26T07:04:03.8631230Z remote: Counting objects: 73% (1891/2590) 2024-06-26T07:04:03.8631809Z remote: Counting objects: 74% (1917/2590) 2024-06-26T07:04:03.8632395Z remote: Counting objects: 75% (1943/2590) 2024-06-26T07:04:03.8632986Z remote: Counting objects: 76% (1969/2590) 2024-06-26T07:04:03.8633571Z remote: Counting objects: 77% (1995/2590) 2024-06-26T07:04:03.8634146Z remote: Counting objects: 78% (2021/2590) 2024-06-26T07:04:03.8634852Z remote: Counting objects: 79% (2047/2590) 2024-06-26T07:04:03.8635451Z remote: Counting objects: 80% (2072/2590) 2024-06-26T07:04:03.8636044Z remote: Counting objects: 81% (2098/2590) 2024-06-26T07:04:03.8636624Z remote: Counting objects: 82% (2124/2590) 2024-06-26T07:04:03.8637210Z remote: Counting objects: 83% (2150/2590) 2024-06-26T07:04:03.8637792Z remote: Counting objects: 84% (2176/2590) 2024-06-26T07:04:03.8638367Z remote: Counting objects: 85% (2202/2590) 2024-06-26T07:04:03.8638953Z remote: Counting objects: 86% (2228/2590) 2024-06-26T07:04:03.8639536Z remote: Counting objects: 87% (2254/2590) 2024-06-26T07:04:03.8640120Z remote: Counting objects: 88% (2280/2590) 2024-06-26T07:04:03.8640731Z remote: Counting objects: 89% (2306/2590) 2024-06-26T07:04:03.8641329Z remote: Counting objects: 90% (2331/2590) 2024-06-26T07:04:03.8641936Z remote: Counting objects: 91% (2357/2590) 2024-06-26T07:04:03.8642608Z remote: Counting objects: 92% (2383/2590) 2024-06-26T07:04:03.8643203Z remote: Counting objects: 93% (2409/2590) 2024-06-26T07:04:03.8643785Z remote: Counting objects: 94% (2435/2590) 2024-06-26T07:04:03.8644377Z remote: Counting objects: 95% (2461/2590) 2024-06-26T07:04:03.8644967Z remote: Counting objects: 96% (2487/2590) 2024-06-26T07:04:03.8645556Z remote: Counting objects: 97% (2513/2590) 2024-06-26T07:04:03.8646137Z remote: Counting objects: 98% (2539/2590) 2024-06-26T07:04:03.8646729Z remote: Counting objects: 99% (2565/2590) 2024-06-26T07:04:03.8647319Z remote: Counting objects: 100% (2590/2590) 2024-06-26T07:04:03.8647934Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T07:04:03.8810179Z remote: Compressing objects: 0% (1/1068) 2024-06-26T07:04:03.9312660Z remote: Compressing objects: 1% (11/1068) 2024-06-26T07:04:03.9749086Z remote: Compressing objects: 2% (22/1068) 2024-06-26T07:04:04.0119433Z remote: Compressing objects: 3% (33/1068) 2024-06-26T07:04:04.0392987Z remote: Compressing objects: 4% (43/1068) 2024-06-26T07:04:04.0678445Z remote: Compressing objects: 5% (54/1068) 2024-06-26T07:04:04.0881882Z remote: Compressing objects: 6% (65/1068) 2024-06-26T07:04:04.1281792Z remote: Compressing objects: 7% (75/1068) 2024-06-26T07:04:04.2228925Z remote: Compressing objects: 8% (86/1068) 2024-06-26T07:04:04.3013756Z remote: Compressing objects: 9% (97/1068) 2024-06-26T07:04:04.3260100Z remote: Compressing objects: 10% (107/1068) 2024-06-26T07:04:04.3443370Z remote: Compressing objects: 11% (118/1068) 2024-06-26T07:04:04.3548367Z remote: Compressing objects: 12% (129/1068) 2024-06-26T07:04:04.3549371Z remote: Compressing objects: 13% (139/1068) 2024-06-26T07:04:04.3675898Z remote: Compressing objects: 14% (150/1068) 2024-06-26T07:04:04.3676467Z remote: Compressing objects: 15% (161/1068) 2024-06-26T07:04:04.3739513Z remote: Compressing objects: 16% (171/1068) 2024-06-26T07:04:04.3740817Z remote: Compressing objects: 17% (182/1068) 2024-06-26T07:04:04.3741691Z remote: Compressing objects: 18% (193/1068) 2024-06-26T07:04:04.3759226Z remote: Compressing objects: 19% (203/1068) 2024-06-26T07:04:04.3776084Z remote: Compressing objects: 20% (214/1068) 2024-06-26T07:04:04.3792118Z remote: Compressing objects: 21% (225/1068) 2024-06-26T07:04:04.3806718Z remote: Compressing objects: 22% (235/1068) 2024-06-26T07:04:04.3830480Z remote: Compressing objects: 23% (246/1068) 2024-06-26T07:04:04.3846065Z remote: Compressing objects: 24% (257/1068) 2024-06-26T07:04:04.3857173Z remote: Compressing objects: 25% (267/1068) 2024-06-26T07:04:04.3864240Z remote: Compressing objects: 26% (278/1068) 2024-06-26T07:04:04.3866373Z remote: Compressing objects: 27% (289/1068) 2024-06-26T07:04:04.3870719Z remote: Compressing objects: 28% (300/1068) 2024-06-26T07:04:04.3879639Z remote: Compressing objects: 29% (310/1068) 2024-06-26T07:04:04.3887240Z remote: Compressing objects: 30% (321/1068) 2024-06-26T07:04:04.3893679Z remote: Compressing objects: 31% (332/1068) 2024-06-26T07:04:04.3900900Z remote: Compressing objects: 32% (342/1068) 2024-06-26T07:04:04.3912061Z remote: Compressing objects: 33% (353/1068) 2024-06-26T07:04:04.3925361Z remote: Compressing objects: 34% (364/1068) 2024-06-26T07:04:04.3942617Z remote: Compressing objects: 35% (374/1068) 2024-06-26T07:04:04.3967611Z remote: Compressing objects: 36% (385/1068) 2024-06-26T07:04:04.3991048Z remote: Compressing objects: 37% (396/1068) 2024-06-26T07:04:04.4001401Z remote: Compressing objects: 38% (406/1068) 2024-06-26T07:04:04.4009393Z remote: Compressing objects: 39% (417/1068) 2024-06-26T07:04:04.4025836Z remote: Compressing objects: 40% (428/1068) 2024-06-26T07:04:04.4045006Z remote: Compressing objects: 41% (438/1068) 2024-06-26T07:04:04.4059770Z remote: Compressing objects: 42% (449/1068) 2024-06-26T07:04:04.4074965Z remote: Compressing objects: 43% (460/1068) 2024-06-26T07:04:04.4079366Z remote: Compressing objects: 44% (470/1068) 2024-06-26T07:04:04.4084552Z remote: Compressing objects: 45% (481/1068) 2024-06-26T07:04:04.4099823Z remote: Compressing objects: 46% (492/1068) 2024-06-26T07:04:04.4113420Z remote: Compressing objects: 47% (502/1068) 2024-06-26T07:04:04.4126248Z remote: Compressing objects: 48% (513/1068) 2024-06-26T07:04:04.4131132Z remote: Compressing objects: 49% (524/1068) 2024-06-26T07:04:04.4140436Z remote: Compressing objects: 50% (534/1068) 2024-06-26T07:04:04.4151916Z remote: Compressing objects: 51% (545/1068) 2024-06-26T07:04:04.4167387Z remote: Compressing objects: 52% (556/1068) 2024-06-26T07:04:04.4176095Z remote: Compressing objects: 53% (567/1068) 2024-06-26T07:04:04.4192429Z remote: Compressing objects: 54% (577/1068) 2024-06-26T07:04:04.4202891Z remote: Compressing objects: 55% (588/1068) 2024-06-26T07:04:04.4210640Z remote: Compressing objects: 56% (599/1068) 2024-06-26T07:04:04.4226536Z remote: Compressing objects: 57% (609/1068) 2024-06-26T07:04:04.4238056Z remote: Compressing objects: 58% (620/1068) 2024-06-26T07:04:04.4245111Z remote: Compressing objects: 59% (631/1068) 2024-06-26T07:04:04.4255825Z remote: Compressing objects: 60% (641/1068) 2024-06-26T07:04:04.4270768Z remote: Compressing objects: 61% (652/1068) 2024-06-26T07:04:04.4280362Z remote: Compressing objects: 62% (663/1068) 2024-06-26T07:04:04.4292742Z remote: Compressing objects: 63% (673/1068) 2024-06-26T07:04:04.4293283Z remote: Compressing objects: 64% (684/1068) 2024-06-26T07:04:04.4301568Z remote: Compressing objects: 65% (695/1068) 2024-06-26T07:04:04.4303179Z remote: Compressing objects: 66% (705/1068) 2024-06-26T07:04:04.4314688Z remote: Compressing objects: 67% (716/1068) 2024-06-26T07:04:04.4318294Z remote: Compressing objects: 68% (727/1068) 2024-06-26T07:04:04.4322011Z remote: Compressing objects: 69% (737/1068) 2024-06-26T07:04:04.4327387Z remote: Compressing objects: 70% (748/1068) 2024-06-26T07:04:04.4328271Z remote: Compressing objects: 71% (759/1068) 2024-06-26T07:04:04.4329300Z remote: Compressing objects: 72% (769/1068) 2024-06-26T07:04:04.4331302Z remote: Compressing objects: 73% (780/1068) 2024-06-26T07:04:04.4332307Z remote: Compressing objects: 74% (791/1068) 2024-06-26T07:04:04.4333263Z remote: Compressing objects: 75% (801/1068) 2024-06-26T07:04:04.4333884Z remote: Compressing objects: 76% (812/1068) 2024-06-26T07:04:04.4334619Z remote: Compressing objects: 77% (823/1068) 2024-06-26T07:04:04.4335238Z remote: Compressing objects: 78% (834/1068) 2024-06-26T07:04:04.4338727Z remote: Compressing objects: 79% (844/1068) 2024-06-26T07:04:04.4339588Z remote: Compressing objects: 80% (855/1068) 2024-06-26T07:04:04.4342041Z remote: Compressing objects: 81% (866/1068) 2024-06-26T07:04:04.4345073Z remote: Compressing objects: 82% (876/1068) 2024-06-26T07:04:04.4348600Z remote: Compressing objects: 83% (887/1068) 2024-06-26T07:04:04.4351901Z remote: Compressing objects: 84% (898/1068) 2024-06-26T07:04:04.4352498Z remote: Compressing objects: 85% (908/1068) 2024-06-26T07:04:04.4353074Z remote: Compressing objects: 86% (919/1068) 2024-06-26T07:04:04.4354807Z remote: Compressing objects: 87% (930/1068) 2024-06-26T07:04:04.4355356Z remote: Compressing objects: 88% (940/1068) 2024-06-26T07:04:04.4357163Z remote: Compressing objects: 89% (951/1068) 2024-06-26T07:04:04.4358769Z remote: Compressing objects: 90% (962/1068) 2024-06-26T07:04:04.4359594Z remote: Compressing objects: 91% (972/1068) 2024-06-26T07:04:04.4361401Z remote: Compressing objects: 92% (983/1068) 2024-06-26T07:04:04.4363256Z remote: Compressing objects: 93% (994/1068) 2024-06-26T07:04:04.4364566Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T07:04:04.4366378Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T07:04:04.4367013Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T07:04:04.4368008Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T07:04:04.4370019Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T07:04:04.4370767Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T07:04:04.4371620Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T07:04:04.4372270Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:04:24.5194206Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:04:44.6393487Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:04:44.6412338Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:44.6415675Z ##[endgroup] 2024-06-26T07:04:44.6416328Z ##[group]Determining the checkout info 2024-06-26T07:04:44.6418637Z ##[endgroup] 2024-06-26T07:04:44.6419309Z ##[group]Checking out the ref 2024-06-26T07:04:44.6420127Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:45.8540133Z ##[endgroup] 2024-06-26T07:04:45.8540818Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:04:45.8545127Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:04:45.8588952Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:04:45.8609195Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:04:45.8634115Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:04:45.8655332Z ##[endgroup] 2024-06-26T07:04:45.8655869Z ##[group]Fetching submodules 2024-06-26T07:04:45.8658907Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:04:45.8869643Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:04:45.9077424Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:04:45.9078834Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:04:45.9080243Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:04:45.9081763Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:04:45.9083529Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:45.9085168Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:04:45.9086627Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:04:45.9088907Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:04:45.9090821Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:04:45.9093256Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:04:45.9095426Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:04:45.9097735Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:04:45.9100082Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:04:45.9102647Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:04:45.9105678Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:04:45.9108165Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:04:45.9110976Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:45.9113515Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:04:45.9116388Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:04:45.9119120Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:04:45.9122154Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:04:45.9125294Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:04:45.9128481Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:04:45.9131519Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:04:45.9134732Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:04:45.9137929Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:04:45.9141524Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:04:45.9145129Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:04:45.9148502Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:04:45.9151869Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:04:45.9155612Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:04:45.9159209Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:04:45.9162979Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:04:45.9166670Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:04:45.9170521Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:04:45.9190922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:04:46.1571307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:04:46.3216890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:04:46.5335265Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:04:46.7751972Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:04:48.7608853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:04:58.5918359Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:04:58.9472600Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:04:59.4535102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:05:00.0127382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:05:01.2090727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:05:02.9043212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:05:08.8890602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:05:10.2044724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:05:11.8200345Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:05:13.0122138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:05:13.1647471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:05:13.5739392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:05:13.8766592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:05:14.8319551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:05:15.1767174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:05:15.4184307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:05:16.9519448Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:05:17.6801286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:05:18.4038896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:05:23.9333046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:05:26.2027149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:05:31.8212772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:05:32.0797850Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:05:40.1222799Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:05:40.2720544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:05:40.4555152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:05:41.4442699Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:05:41.7042238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:05:42.3153230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:05:42.7135116Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:05:42.7219379Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:05:42.7279504Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:05:42.7460246Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:05:42.7760686Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:05:43.5498014Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:05:43.5670164Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:05:43.6012180Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:05:43.6856552Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:05:43.7126488Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:05:44.1397179Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:05:44.3512138Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:05:44.4140236Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:05:44.4151748Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:44.4153353Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:44.4155234Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:44.4157340Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:44.4159331Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:44.4174879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:05:45.4061112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:05:45.9637626Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:05:47.7061416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:05:48.6308968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:05:48.9320051Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:05:49.0159146Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:05:49.3617328Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:05:49.4159327Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:05:49.4243268Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:05:49.5153451Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:05:49.5434500Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:05:49.5499868Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:05:49.5805079Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:05:49.5999455Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:05:49.6381129Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:05:49.6477152Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:05:49.6488414Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:49.6505510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:06:01.8280687Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:06:01.8421257Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:06:01.9200509Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:06:01.9212911Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:01.9214737Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:01.9217593Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:01.9233231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:06:02.4751234Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:06:03.7343791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:06:04.7898552Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:06:04.7908258Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:04.7909924Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:04.7911789Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:04.7913606Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:04.7915463Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:04.7917810Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:04.7919745Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:04.7922047Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:04.7940628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:06:05.8424433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:06:06.1962144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:06:07.4438200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:06:07.7201982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:06:08.2213965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:06:09.2761820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:06:14.8069369Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:06:15.1466281Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:06:15.1607486Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:06:15.1903647Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:06:15.2000796Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:06:15.2011611Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:15.2028348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:06:15.5655620Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:06:15.5792433Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:06:15.6125706Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:06:15.6919266Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:06:15.7041878Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:06:15.7338078Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:06:15.7830688Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:06:15.8141727Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:06:15.8338348Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:06:15.9208420Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:06:16.2003593Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:06:16.2026996Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:16.2028803Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:16.2044699Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:06:16.6143663Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:06:17.7895342Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:06:17.8140756Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:06:17.8668212Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:06:17.8679576Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:17.8687132Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:17.8688815Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:17.8690697Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:17.8692643Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:17.8694725Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:17.8696673Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:17.8698338Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:17.8712172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:06:18.3145900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:06:19.3330956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:06:19.6261439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:06:25.6396665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:06:25.8985501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:06:26.1007218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:06:26.3833400Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:06:33.8160800Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:06:33.8492165Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:06:33.8616492Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:06:33.9452989Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:06:33.9545099Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:06:33.9651758Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:06:33.9763865Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:06:33.9773942Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:33.9776110Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:33.9792724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:06:35.8066629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:06:36.9930611Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:06:37.0297447Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:06:37.3899732Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:06:37.3980803Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:06:37.6123876Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:06:37.6138907Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:37.6140952Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:37.6158490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:06:37.9910987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:06:38.9961003Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:06:39.0584632Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:06:39.0646141Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:06:39.0734348Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:06:39.1012705Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:06:39.1237217Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:06:39.1566501Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:06:39.1769001Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:06:39.1780352Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:39.1782284Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:39.1784487Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:39.1787139Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:39.1804971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:06:40.1472388Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:06:40.3637187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:06:41.3612537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:06:42.4085018Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:06:42.4195985Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:06:42.4762483Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:06:42.4989805Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:06:42.4998603Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:42.5013385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:06:42.7447221Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:06:42.7470734Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:06:42.7675956Z Entering 'android/libs/fbjni' 2024-06-26T07:06:42.7704017Z Entering 'third_party/FP16' 2024-06-26T07:06:42.7730048Z Entering 'third_party/FXdiv' 2024-06-26T07:06:42.7757275Z Entering 'third_party/NNPACK' 2024-06-26T07:06:42.7782834Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:42.7809181Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:42.7848520Z Entering 'third_party/benchmark' 2024-06-26T07:06:42.7877224Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:42.7906051Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:42.7931668Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:42.7957950Z Entering 'third_party/cutlass' 2024-06-26T07:06:42.7994462Z Entering 'third_party/eigen' 2024-06-26T07:06:42.8021623Z Entering 'third_party/fbgemm' 2024-06-26T07:06:42.8050735Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:42.8076940Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:42.8103118Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:42.8133088Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:42.8160896Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:42.8187730Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:42.8219292Z Entering 'third_party/fmt' 2024-06-26T07:06:42.8248801Z Entering 'third_party/foxi' 2024-06-26T07:06:42.8277062Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:42.8304270Z Entering 'third_party/gloo' 2024-06-26T07:06:42.8333082Z Entering 'third_party/googletest' 2024-06-26T07:06:42.8359588Z Entering 'third_party/ideep' 2024-06-26T07:06:42.8385885Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:42.8418268Z Entering 'third_party/ittapi' 2024-06-26T07:06:42.8446364Z Entering 'third_party/kineto' 2024-06-26T07:06:42.8471373Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:42.8499231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:42.8527576Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:42.8553146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:42.8581295Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:42.8608184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:42.8635525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:42.8664253Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:42.8692190Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:42.8719893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:42.8750076Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:42.8777480Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:42.8806863Z Entering 'third_party/mimalloc' 2024-06-26T07:06:42.8834111Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:42.8861358Z Entering 'third_party/nlohmann' 2024-06-26T07:06:42.8889024Z Entering 'third_party/onnx' 2024-06-26T07:06:42.8929094Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:42.8958203Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:42.8986372Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:42.9016256Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:42.9042454Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:42.9069554Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:42.9096899Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:42.9126484Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:42.9151272Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:42.9177791Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:42.9203214Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:42.9231046Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:42.9260590Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:42.9304847Z Entering 'third_party/pocketfft' 2024-06-26T07:06:42.9333943Z Entering 'third_party/protobuf' 2024-06-26T07:06:42.9364465Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:42.9392336Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:42.9421135Z Entering 'third_party/psimd' 2024-06-26T07:06:42.9447281Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:42.9475846Z Entering 'third_party/pybind11' 2024-06-26T07:06:42.9504078Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:42.9531519Z Entering 'third_party/sleef' 2024-06-26T07:06:42.9559197Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:42.9585038Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:42.9611558Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:42.9636812Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:42.9662566Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:42.9691264Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:42.9727674Z ##[endgroup] 2024-06-26T07:06:42.9729754Z ##[group]Persisting credentials for submodules 2024-06-26T07:06:42.9732803Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:06:42.9943037Z Entering 'android/libs/fbjni' 2024-06-26T07:06:42.9978266Z Entering 'third_party/FP16' 2024-06-26T07:06:43.0014648Z Entering 'third_party/FXdiv' 2024-06-26T07:06:43.0050963Z Entering 'third_party/NNPACK' 2024-06-26T07:06:43.0084615Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:43.0120535Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:43.0163782Z Entering 'third_party/benchmark' 2024-06-26T07:06:43.0199670Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:43.0233609Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:43.0269284Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:43.0306236Z Entering 'third_party/cutlass' 2024-06-26T07:06:43.0348075Z Entering 'third_party/eigen' 2024-06-26T07:06:43.0384547Z Entering 'third_party/fbgemm' 2024-06-26T07:06:43.0420117Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:43.0454521Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:43.0488800Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:43.0528728Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:43.0564253Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:43.0597183Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:43.0635293Z Entering 'third_party/fmt' 2024-06-26T07:06:43.0670803Z Entering 'third_party/foxi' 2024-06-26T07:06:43.0705389Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:43.0739072Z Entering 'third_party/gloo' 2024-06-26T07:06:43.0770753Z Entering 'third_party/googletest' 2024-06-26T07:06:43.0803935Z Entering 'third_party/ideep' 2024-06-26T07:06:43.0836740Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:43.0873866Z Entering 'third_party/ittapi' 2024-06-26T07:06:43.0907761Z Entering 'third_party/kineto' 2024-06-26T07:06:43.0942163Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:43.0977148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:43.1012894Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:43.1046425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:43.1081395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:43.1116037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:43.1152101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:43.1184874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:43.1217949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:43.1252557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:43.1287958Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:43.1321390Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:43.1356517Z Entering 'third_party/mimalloc' 2024-06-26T07:06:43.1392291Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:43.1425998Z Entering 'third_party/nlohmann' 2024-06-26T07:06:43.1461365Z Entering 'third_party/onnx' 2024-06-26T07:06:43.1510901Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:43.1545040Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:43.1578684Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:43.1615754Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:43.1648798Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:43.1681693Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:43.1716357Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:43.1750257Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:43.1785856Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:43.1817309Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:43.1851317Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:43.1885041Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:43.1919991Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:43.1966271Z Entering 'third_party/pocketfft' 2024-06-26T07:06:43.2001131Z Entering 'third_party/protobuf' 2024-06-26T07:06:43.2035453Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:43.2073602Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:43.2109361Z Entering 'third_party/psimd' 2024-06-26T07:06:43.2142936Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:43.2176834Z Entering 'third_party/pybind11' 2024-06-26T07:06:43.2209957Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:43.2245433Z Entering 'third_party/sleef' 2024-06-26T07:06:43.2281082Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:43.2316528Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:43.2348634Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:43.2381651Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:43.2416260Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:43.2449078Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:43.2498618Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:06:43.2703130Z Entering 'android/libs/fbjni' 2024-06-26T07:06:43.2735506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:06:43.2748822Z Entering 'third_party/FP16' 2024-06-26T07:06:43.2779816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:06:43.2793432Z Entering 'third_party/FXdiv' 2024-06-26T07:06:43.2822996Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:06:43.2835376Z Entering 'third_party/NNPACK' 2024-06-26T07:06:43.2868846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:06:43.2883026Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:43.2915916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:06:43.2928497Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:43.2960189Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:06:43.2982396Z Entering 'third_party/benchmark' 2024-06-26T07:06:43.3014710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:43.3028896Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:43.3062182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:06:43.3076473Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:43.3108802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:06:43.3123175Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:43.3154670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:06:43.3168643Z Entering 'third_party/cutlass' 2024-06-26T07:06:43.3200151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:06:43.3218038Z Entering 'third_party/eigen' 2024-06-26T07:06:43.3251814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:06:43.3265711Z Entering 'third_party/fbgemm' 2024-06-26T07:06:43.3297079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:06:43.3309070Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:43.3340284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:06:43.3353200Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:43.3385061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:06:43.3398485Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:43.3429066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:06:43.3443888Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:43.3475995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:43.3489083Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:43.3520690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:06:43.3533856Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:43.3565510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:06:43.3581827Z Entering 'third_party/fmt' 2024-06-26T07:06:43.3614378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:06:43.3627665Z Entering 'third_party/foxi' 2024-06-26T07:06:43.3658721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:06:43.3671512Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:43.3700938Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:06:43.3712855Z Entering 'third_party/gloo' 2024-06-26T07:06:43.3743963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:06:43.3756115Z Entering 'third_party/googletest' 2024-06-26T07:06:43.3787826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:43.3801055Z Entering 'third_party/ideep' 2024-06-26T07:06:43.3833136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:06:43.3845470Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:43.3875155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:06:43.3891498Z Entering 'third_party/ittapi' 2024-06-26T07:06:43.3923401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:06:43.3935087Z Entering 'third_party/kineto' 2024-06-26T07:06:43.3965396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:06:43.3977037Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:43.4011430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:06:43.4022504Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:43.4056955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:06:43.4070555Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:43.4101605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:06:43.4122668Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:43.4149052Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:06:43.4161677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:43.4192973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:06:43.4205375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:43.4237516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:06:43.4252431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:43.4284882Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:06:43.4296941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:43.4329507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:43.4341510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:43.4374879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:06:43.4387808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:43.4419345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:06:43.4432843Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:43.4464501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:06:43.4477714Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:43.4508996Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:06:43.4523216Z Entering 'third_party/mimalloc' 2024-06-26T07:06:43.4555719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:06:43.4568735Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:43.4599757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:06:43.4612649Z Entering 'third_party/nlohmann' 2024-06-26T07:06:43.4645717Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:06:43.4659411Z Entering 'third_party/onnx' 2024-06-26T07:06:43.4691601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:06:43.4715686Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:43.4748607Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:43.4761377Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:43.4793497Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:43.4808316Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:43.4840985Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:06:43.4854575Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:43.4887234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:43.4899889Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:43.4929345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:43.4940163Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:43.4972814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:06:43.4985953Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:43.5017429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:06:43.5031559Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:43.5064154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:06:43.5076630Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:43.5108293Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:06:43.5120175Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:43.5152697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:06:43.5164174Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:43.5195379Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:06:43.5210405Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:43.5243017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:06:43.5255518Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:43.5288176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:06:43.5316268Z Entering 'third_party/pocketfft' 2024-06-26T07:06:43.5346870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:06:43.5358852Z Entering 'third_party/protobuf' 2024-06-26T07:06:43.5390563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:06:43.5405714Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:43.5437300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:06:43.5449631Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:43.5481156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:43.5494884Z Entering 'third_party/psimd' 2024-06-26T07:06:43.5527321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:06:43.5539154Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:43.5571222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:06:43.5585257Z Entering 'third_party/pybind11' 2024-06-26T07:06:43.5616491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:43.5630787Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:43.5661738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:06:43.5673728Z Entering 'third_party/sleef' 2024-06-26T07:06:43.5704711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:06:43.5716908Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:43.5749374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:06:43.5761248Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:43.5793018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:06:43.5805918Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:43.5835848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:06:43.5849151Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:43.5878981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:06:43.5889996Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:43.5921634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:06:43.5932781Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:43.5964538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:06:43.6780618Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:06:43.6982481Z Entering 'android/libs/fbjni' 2024-06-26T07:06:43.7008277Z Entering 'third_party/FP16' 2024-06-26T07:06:43.7033172Z Entering 'third_party/FXdiv' 2024-06-26T07:06:43.7060416Z Entering 'third_party/NNPACK' 2024-06-26T07:06:43.7088138Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:43.7113706Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:43.7149970Z Entering 'third_party/benchmark' 2024-06-26T07:06:43.7176273Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:43.7201957Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:43.7232918Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:43.7262227Z Entering 'third_party/cutlass' 2024-06-26T07:06:43.7295408Z Entering 'third_party/eigen' 2024-06-26T07:06:43.7324777Z Entering 'third_party/fbgemm' 2024-06-26T07:06:43.7351091Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:43.7378328Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:43.7406612Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:43.7437823Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:43.7464309Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:43.7492876Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:43.7524052Z Entering 'third_party/fmt' 2024-06-26T07:06:43.7552010Z Entering 'third_party/foxi' 2024-06-26T07:06:43.7579034Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:43.7609267Z Entering 'third_party/gloo' 2024-06-26T07:06:43.7635354Z Entering 'third_party/googletest' 2024-06-26T07:06:43.7664929Z Entering 'third_party/ideep' 2024-06-26T07:06:43.7691209Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:43.7723448Z Entering 'third_party/ittapi' 2024-06-26T07:06:43.7752469Z Entering 'third_party/kineto' 2024-06-26T07:06:43.7779196Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:43.7805996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:43.7833928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:43.7859676Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:43.7887579Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:43.7913670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:43.7942005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:43.7970308Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:43.7995915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:43.8024004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:43.8052883Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:43.8079378Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:43.8111920Z Entering 'third_party/mimalloc' 2024-06-26T07:06:43.8139563Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:43.8168193Z Entering 'third_party/nlohmann' 2024-06-26T07:06:43.8195505Z Entering 'third_party/onnx' 2024-06-26T07:06:43.8236043Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:43.8266227Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:43.8297918Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:43.8328408Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:43.8354332Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:43.8381326Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:43.8409783Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:43.8439265Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:43.8467597Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:43.8495159Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:43.8520631Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:43.8550275Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:43.8578716Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:43.8621410Z Entering 'third_party/pocketfft' 2024-06-26T07:06:43.8649543Z Entering 'third_party/protobuf' 2024-06-26T07:06:43.8679310Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:43.8708176Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:43.8735527Z Entering 'third_party/psimd' 2024-06-26T07:06:43.8763580Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:43.8793100Z Entering 'third_party/pybind11' 2024-06-26T07:06:43.8820942Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:43.8850638Z Entering 'third_party/sleef' 2024-06-26T07:06:43.8879446Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:43.8906821Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:43.8934014Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:43.8960282Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:43.8986490Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:43.9011870Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:43.9053770Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:06:43.9264707Z Entering 'android/libs/fbjni' 2024-06-26T07:06:43.9293152Z Entering 'third_party/FP16' 2024-06-26T07:06:43.9322700Z Entering 'third_party/FXdiv' 2024-06-26T07:06:43.9351483Z Entering 'third_party/NNPACK' 2024-06-26T07:06:43.9379617Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:06:43.9406777Z Entering 'third_party/XNNPACK' 2024-06-26T07:06:43.9446872Z Entering 'third_party/benchmark' 2024-06-26T07:06:43.9477089Z Entering 'third_party/cpp-httplib' 2024-06-26T07:06:43.9506828Z Entering 'third_party/cpuinfo' 2024-06-26T07:06:43.9536377Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:06:43.9563693Z Entering 'third_party/cutlass' 2024-06-26T07:06:43.9595348Z Entering 'third_party/eigen' 2024-06-26T07:06:43.9625836Z Entering 'third_party/fbgemm' 2024-06-26T07:06:43.9654146Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:43.9683569Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:43.9710606Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:43.9742070Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:43.9769927Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:43.9797534Z Entering 'third_party/flatbuffers' 2024-06-26T07:06:43.9829231Z Entering 'third_party/fmt' 2024-06-26T07:06:43.9857831Z Entering 'third_party/foxi' 2024-06-26T07:06:43.9886820Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:06:43.9913366Z Entering 'third_party/gloo' 2024-06-26T07:06:43.9941808Z Entering 'third_party/googletest' 2024-06-26T07:06:43.9969840Z Entering 'third_party/ideep' 2024-06-26T07:06:43.9997314Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:44.0028624Z Entering 'third_party/ittapi' 2024-06-26T07:06:44.0057825Z Entering 'third_party/kineto' 2024-06-26T07:06:44.0086965Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:06:44.0113183Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:06:44.0142586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:06:44.0168830Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:06:44.0195862Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:06:44.0222359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:06:44.0251091Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:06:44.0279208Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:06:44.0306639Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:06:44.0335487Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:06:44.0364278Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:06:44.0390513Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:06:44.0419301Z Entering 'third_party/mimalloc' 2024-06-26T07:06:44.0449571Z Entering 'third_party/nccl/nccl' 2024-06-26T07:06:44.0477264Z Entering 'third_party/nlohmann' 2024-06-26T07:06:44.0506491Z Entering 'third_party/onnx' 2024-06-26T07:06:44.0545284Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:06:44.0573431Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:06:44.0602313Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:06:44.0633688Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:06:44.0661564Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:06:44.0690295Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:06:44.0717671Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:06:44.0744699Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:06:44.0772600Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:06:44.0800542Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:06:44.0826377Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:06:44.0856212Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:06:44.0883912Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:06:44.0927168Z Entering 'third_party/pocketfft' 2024-06-26T07:06:44.0957052Z Entering 'third_party/protobuf' 2024-06-26T07:06:44.0985359Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:06:44.1013567Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:06:44.1044033Z Entering 'third_party/psimd' 2024-06-26T07:06:44.1072395Z Entering 'third_party/pthreadpool' 2024-06-26T07:06:44.1100740Z Entering 'third_party/pybind11' 2024-06-26T07:06:44.1130178Z Entering 'third_party/python-peachpy' 2024-06-26T07:06:44.1158359Z Entering 'third_party/sleef' 2024-06-26T07:06:44.1187107Z Entering 'third_party/tensorpipe' 2024-06-26T07:06:44.1215370Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:06:44.1242006Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:06:44.1269696Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:06:44.1298176Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:06:44.1325162Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:06:44.1363239Z ##[endgroup] 2024-06-26T07:06:44.1392300Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:06:44.1413681Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:06:44.1583131Z Prepare all required actions 2024-06-26T07:06:44.1583820Z Getting action download info 2024-06-26T07:06:44.3004698Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:06:44.3005120Z env: 2024-06-26T07:06:44.3005398Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:44.3005744Z ##[endgroup] 2024-06-26T07:06:44.3063556Z ##[group]Run set -euo pipefail 2024-06-26T07:06:44.3063999Z set -euo pipefail 2024-06-26T07:06:44.3064383Z function get_ec2_metadata() { 2024-06-26T07:06:44.3064913Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:06:44.3065786Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:06:44.3066546Z  category=$1 2024-06-26T07:06:44.3067065Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:06:44.3067693Z  runner_name_str=i-08e7836a94a47ea58 2024-06-26T07:06:44.3068184Z  if [[ -f /.inarc ]]; then 2024-06-26T07:06:44.3068668Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:06:44.3069232Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:06:44.3069921Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:06:44.3070525Z  else 2024-06-26T07:06:44.3071007Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:06:44.3071580Z  fi 2024-06-26T07:06:44.3071868Z } 2024-06-26T07:06:44.3072228Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:06:44.3072798Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:06:44.3073443Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:06:44.3074005Z echo "system info $(uname -a)" 2024-06-26T07:06:44.3081780Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:44.3082333Z env: 2024-06-26T07:06:44.3082617Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:44.3082964Z ##[endgroup] 2024-06-26T07:06:44.3150352Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:06:44.3196969Z instance-id: i-08e7836a94a47ea58 2024-06-26T07:06:44.3240979Z instance-type: g5.4xlarge 2024-06-26T07:06:44.3247118Z system info Linux ip-10-0-2-254.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:06:44.3268228Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:06:44.3269139Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:06:44.3277084Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:44.3277576Z env: 2024-06-26T07:06:44.3277860Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:44.3278209Z ##[endgroup] 2024-06-26T07:06:44.3338974Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:06:44.3339586Z if systemctl is-active --quiet docker; then 2024-06-26T07:06:44.3340121Z  echo "Docker daemon is running..."; 2024-06-26T07:06:44.3340714Z else 2024-06-26T07:06:44.3341209Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:06:44.3341790Z fi 2024-06-26T07:06:44.3349653Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:44.3350154Z env: 2024-06-26T07:06:44.3350424Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:44.3350772Z ##[endgroup] 2024-06-26T07:06:44.3388837Z Docker daemon is running... 2024-06-26T07:06:44.3431186Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:06:44.3431739Z with: 2024-06-26T07:06:44.3432006Z shell: bash 2024-06-26T07:06:44.3432298Z timeout_minutes: 5 2024-06-26T07:06:44.3432620Z max_attempts: 3 2024-06-26T07:06:44.3432936Z retry_wait_seconds: 30 2024-06-26T07:06:44.3434432Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:06:44.3435860Z polling_interval_seconds: 1 2024-06-26T07:06:44.3436225Z warning_on_retry: true 2024-06-26T07:06:44.3436570Z continue_on_error: false 2024-06-26T07:06:44.3436904Z env: 2024-06-26T07:06:44.3437169Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:44.3437522Z AWS_RETRY_MODE: standard 2024-06-26T07:06:44.3437874Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:06:44.3438215Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:06:44.3438578Z ##[endgroup] 2024-06-26T07:06:45.2709122Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:45.2710142Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:45.2710981Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:45.2711493Z 2024-06-26T07:06:45.2711647Z Login Succeeded 2024-06-26T07:06:45.3922842Z Command completed after 1 attempt(s). 2024-06-26T07:06:45.3963872Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:45.3964599Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:45.3965251Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:06:45.3973071Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:45.3973566Z env: 2024-06-26T07:06:45.3973847Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:45.3974196Z ##[endgroup] 2024-06-26T07:06:45.4032029Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:06:45.4032785Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:06:45.4033370Z # shellcheck disable=SC2046 2024-06-26T07:06:45.4033820Z docker stop $(docker ps -q) || true 2024-06-26T07:06:45.4034301Z # Prune all of the docker images 2024-06-26T07:06:45.4034743Z docker system prune -af 2024-06-26T07:06:45.4041808Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:45.4042391Z env: 2024-06-26T07:06:45.4042662Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:45.4043008Z ##[endgroup] 2024-06-26T07:06:45.4284840Z "docker stop" requires at least 1 argument. 2024-06-26T07:06:45.4285693Z See 'docker stop --help'. 2024-06-26T07:06:45.4285931Z 2024-06-26T07:06:45.4286151Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:06:45.4286525Z 2024-06-26T07:06:45.4286722Z Stop one or more running containers 2024-06-26T07:06:45.4433922Z Total reclaimed space: 0B 2024-06-26T07:06:45.4462976Z ##[group]Run set +e 2024-06-26T07:06:45.4463638Z set +e 2024-06-26T07:06:45.4463955Z set -x 2024-06-26T07:06:45.4464254Z  2024-06-26T07:06:45.4464578Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:06:45.4465374Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:06:45.4466444Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:06:45.4467333Z # one is returned at random 2024-06-26T07:06:45.4467868Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:06:45.4468370Z  2024-06-26T07:06:45.4468688Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:06:45.4469297Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:06:45.4470099Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:06:45.4470644Z  2024-06-26T07:06:45.4470971Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:06:45.4471520Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:06:45.4472023Z  exit 1 2024-06-26T07:06:45.4472330Z  fi 2024-06-26T07:06:45.4472621Z fi 2024-06-26T07:06:45.4473014Z  2024-06-26T07:06:45.4473381Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:06:45.4473899Z  # Clean up any old records first 2024-06-26T07:06:45.4474399Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:06:45.4474846Z fi 2024-06-26T07:06:45.4475115Z  2024-06-26T07:06:45.4475539Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:06:45.4476074Z cat /etc/hosts 2024-06-26T07:06:45.4483880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:45.4484372Z env: 2024-06-26T07:06:45.4484660Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:45.4485006Z ##[endgroup] 2024-06-26T07:06:45.4504809Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:06:45.4508867Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:06:45.4509472Z ++ tail -n1 2024-06-26T07:06:45.4754515Z + RESOLVED_IP=18.67.76.99 2024-06-26T07:06:45.4754980Z + '[' -z 18.67.76.99 ']' 2024-06-26T07:06:45.4755428Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:06:45.4760058Z 18.67.76.121 download.pytorch.org 2024-06-26T07:06:45.4760838Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:06:45.4861764Z + echo '18.67.76.99 download.pytorch.org' 2024-06-26T07:06:45.4862488Z + sudo tee -a /etc/hosts 2024-06-26T07:06:45.5215001Z 18.67.76.99 download.pytorch.org 2024-06-26T07:06:45.5227594Z + cat /etc/hosts 2024-06-26T07:06:45.5232325Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:06:45.5245065Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:06:45.5245571Z 18.67.76.99 download.pytorch.org 2024-06-26T07:06:45.5393599Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:06:45.5394664Z with: 2024-06-26T07:06:45.5395859Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5397115Z docker-build-dir: .ci/docker 2024-06-26T07:06:45.5453677Z working-directory: . 2024-06-26T07:06:45.5454233Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:45.5454763Z force-push: false 2024-06-26T07:06:45.5455060Z env: 2024-06-26T07:06:45.5455327Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:45.5455664Z ##[endgroup] 2024-06-26T07:06:45.5475409Z ##[group]Run set -ex 2024-06-26T07:06:45.5475768Z set -ex 2024-06-26T07:06:45.5476060Z  2024-06-26T07:06:45.5476607Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:06:45.5477556Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:06:45.5478352Z # job could then download the pre-built image as usual 2024-06-26T07:06:45.5479080Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:06:45.5480021Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5480649Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5481317Z  2024-06-26T07:06:45.5481819Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:06:45.5482531Z  exit 0 2024-06-26T07:06:45.5482828Z else 2024-06-26T07:06:45.5483188Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5483625Z fi 2024-06-26T07:06:45.5483895Z  2024-06-26T07:06:45.5484355Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:06:45.5485186Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:06:45.5485941Z  # use it as it is, but first let's extract the tag 2024-06-26T07:06:45.5486627Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:06:45.5487328Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5488014Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5488566Z else 2024-06-26T07:06:45.5488987Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:06:45.5489690Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5490563Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5491293Z fi 2024-06-26T07:06:45.5498898Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:45.5499396Z env: 2024-06-26T07:06:45.5499677Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:45.5500021Z REPO_NAME: pytorch 2024-06-26T07:06:45.5501126Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5502300Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:06:45.5502782Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:45.5503486Z ##[endgroup] 2024-06-26T07:06:45.5524094Z + [[ ! -d .ci/docker ]] 2024-06-26T07:06:45.5524496Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:06:45.5524873Z + echo skip=false 2024-06-26T07:06:45.5526761Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:06:45.5529910Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5531135Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:06:45.5536853Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5537508Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5538991Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5565767Z ##[group]Run set +e 2024-06-26T07:06:45.5566132Z set +e 2024-06-26T07:06:45.5566432Z set -x 2024-06-26T07:06:45.5566714Z  2024-06-26T07:06:45.5566992Z login() { 2024-06-26T07:06:45.5567655Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:06:45.5568359Z } 2024-06-26T07:06:45.5568626Z  2024-06-26T07:06:45.5568901Z retry () { 2024-06-26T07:06:45.5569288Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:06:45.5569784Z } 2024-06-26T07:06:45.5570051Z  2024-06-26T07:06:45.5570367Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:06:45.5570777Z  2024-06-26T07:06:45.5571239Z # Check if image already exists, if it does then skip building it 2024-06-26T07:06:45.5572070Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:06:45.5572572Z  exit 0 2024-06-26T07:06:45.5572870Z fi 2024-06-26T07:06:45.5573140Z  2024-06-26T07:06:45.5573630Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:06:45.5574443Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:06:45.5575167Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:06:45.5575821Z  # if we're on the base branch then use the parent commit 2024-06-26T07:06:45.5576409Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:06:45.5576841Z else 2024-06-26T07:06:45.5577301Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:06:45.5577973Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:06:45.5578479Z fi 2024-06-26T07:06:45.5578756Z  2024-06-26T07:06:45.5579072Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:06:45.5579611Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5580056Z  2024-06-26T07:06:45.5580702Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:06:45.5581446Z  exit 0 2024-06-26T07:06:45.5581736Z fi 2024-06-26T07:06:45.5582011Z  2024-06-26T07:06:45.5582441Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:06:45.5583639Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:06:45.5584429Z  exit 1 2024-06-26T07:06:45.5584729Z fi 2024-06-26T07:06:45.5585004Z  2024-06-26T07:06:45.5585484Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:06:45.5586410Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:06:45.5587264Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:06:45.5588211Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:06:45.5589290Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:06:45.5590071Z fi 2024-06-26T07:06:45.5590348Z  2024-06-26T07:06:45.5590701Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:06:45.5598493Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:45.5598983Z env: 2024-06-26T07:06:45.5599253Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:45.5599672Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:06:45.5600133Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:06:45.5601374Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5602693Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:45.5603287Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:45.5603808Z ##[endgroup] 2024-06-26T07:06:45.5622607Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:45.5623251Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:45.5624585Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:06:45.5625363Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:45.9718934Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:45.9719773Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:45.9720578Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:45.9721381Z 2024-06-26T07:06:45.9721505Z Login Succeeded 2024-06-26T07:06:45.9731055Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.1960112Z { 2024-06-26T07:06:46.1960658Z "schemaVersion": 2, 2024-06-26T07:06:46.1961419Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:06:46.1962318Z "config": { 2024-06-26T07:06:46.1962813Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:06:46.1963349Z "size": 48681, 2024-06-26T07:06:46.1963893Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:06:46.1964499Z }, 2024-06-26T07:06:46.1964754Z "layers": [ 2024-06-26T07:06:46.1965030Z { 2024-06-26T07:06:46.1965460Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1966022Z "size": 28580681, 2024-06-26T07:06:46.1966572Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:06:46.1967184Z }, 2024-06-26T07:06:46.1967449Z { 2024-06-26T07:06:46.1968011Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1968712Z "size": 7943451, 2024-06-26T07:06:46.1969422Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:06:46.1970309Z }, 2024-06-26T07:06:46.1970632Z { 2024-06-26T07:06:46.1971102Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1971663Z "size": 55728572, 2024-06-26T07:06:46.1972214Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:06:46.1973175Z }, 2024-06-26T07:06:46.1973512Z { 2024-06-26T07:06:46.1974083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1974826Z "size": 186, 2024-06-26T07:06:46.1975504Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:06:46.1976280Z }, 2024-06-26T07:06:46.1976599Z { 2024-06-26T07:06:46.1977143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1977881Z "size": 6886, 2024-06-26T07:06:46.1978623Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:06:46.1979246Z }, 2024-06-26T07:06:46.1979732Z { 2024-06-26T07:06:46.1980165Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1980728Z "size": 1329001756, 2024-06-26T07:06:46.1981307Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:06:46.1981937Z }, 2024-06-26T07:06:46.1982179Z { 2024-06-26T07:06:46.1982627Z + exit 0 2024-06-26T07:06:46.1983080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1983995Z "size": 62501, 2024-06-26T07:06:46.1984531Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:06:46.1985135Z }, 2024-06-26T07:06:46.1985383Z { 2024-06-26T07:06:46.1985813Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1986350Z "size": 1684, 2024-06-26T07:06:46.1986873Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:06:46.1987465Z }, 2024-06-26T07:06:46.1987716Z { 2024-06-26T07:06:46.1988142Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1988686Z "size": 1523, 2024-06-26T07:06:46.1989238Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:06:46.1989853Z }, 2024-06-26T07:06:46.1990103Z { 2024-06-26T07:06:46.1990529Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1991075Z "size": 2528295403, 2024-06-26T07:06:46.1991749Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:06:46.1992365Z }, 2024-06-26T07:06:46.1992615Z { 2024-06-26T07:06:46.1993035Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1993582Z "size": 86016, 2024-06-26T07:06:46.1994131Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:06:46.1994745Z }, 2024-06-26T07:06:46.1994988Z { 2024-06-26T07:06:46.1995424Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1995968Z "size": 1823, 2024-06-26T07:06:46.1996496Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:06:46.1997090Z }, 2024-06-26T07:06:46.1997343Z { 2024-06-26T07:06:46.1997774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.1998329Z "size": 246737542, 2024-06-26T07:06:46.1998872Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:06:46.1999471Z }, 2024-06-26T07:06:46.1999723Z { 2024-06-26T07:06:46.2000146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2000696Z "size": 545, 2024-06-26T07:06:46.2001246Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:06:46.2001866Z }, 2024-06-26T07:06:46.2002109Z { 2024-06-26T07:06:46.2002670Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2003217Z "size": 1283, 2024-06-26T07:06:46.2003745Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:06:46.2004330Z }, 2024-06-26T07:06:46.2004583Z { 2024-06-26T07:06:46.2005013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2005549Z "size": 485, 2024-06-26T07:06:46.2006099Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:06:46.2006719Z }, 2024-06-26T07:06:46.2006967Z { 2024-06-26T07:06:46.2007388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2007936Z "size": 91710937, 2024-06-26T07:06:46.2008499Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:06:46.2009115Z }, 2024-06-26T07:06:46.2009357Z { 2024-06-26T07:06:46.2009835Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2010466Z "size": 3233, 2024-06-26T07:06:46.2011014Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:06:46.2011620Z }, 2024-06-26T07:06:46.2011866Z { 2024-06-26T07:06:46.2012290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2012826Z "size": 1905, 2024-06-26T07:06:46.2013370Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:06:46.2013996Z }, 2024-06-26T07:06:46.2014246Z { 2024-06-26T07:06:46.2014663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2015215Z "size": 702, 2024-06-26T07:06:46.2015749Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:46.2016356Z }, 2024-06-26T07:06:46.2016599Z { 2024-06-26T07:06:46.2017023Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2017569Z "size": 2784789002, 2024-06-26T07:06:46.2018118Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:06:46.2018719Z }, 2024-06-26T07:06:46.2018970Z { 2024-06-26T07:06:46.2019399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2019987Z "size": 381, 2024-06-26T07:06:46.2020515Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:06:46.2021126Z }, 2024-06-26T07:06:46.2021443Z { 2024-06-26T07:06:46.2021864Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2022407Z "size": 11879, 2024-06-26T07:06:46.2022939Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:06:46.2023747Z }, 2024-06-26T07:06:46.2023990Z { 2024-06-26T07:06:46.2024417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2024958Z "size": 803, 2024-06-26T07:06:46.2025495Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:06:46.2026103Z }, 2024-06-26T07:06:46.2026355Z { 2024-06-26T07:06:46.2026780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2027312Z "size": 106, 2024-06-26T07:06:46.2027832Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:06:46.2028426Z }, 2024-06-26T07:06:46.2028675Z { 2024-06-26T07:06:46.2029099Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2029693Z "size": 503, 2024-06-26T07:06:46.2030218Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:06:46.2030830Z }, 2024-06-26T07:06:46.2031072Z { 2024-06-26T07:06:46.2031497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2032042Z "size": 121477177, 2024-06-26T07:06:46.2032590Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:06:46.2033197Z }, 2024-06-26T07:06:46.2033446Z { 2024-06-26T07:06:46.2033874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2034411Z "size": 109, 2024-06-26T07:06:46.2034938Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:06:46.2035540Z }, 2024-06-26T07:06:46.2035788Z { 2024-06-26T07:06:46.2036203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2036757Z "size": 491, 2024-06-26T07:06:46.2037280Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:06:46.2037867Z }, 2024-06-26T07:06:46.2038119Z { 2024-06-26T07:06:46.2038540Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2039084Z "size": 291, 2024-06-26T07:06:46.2039619Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:06:46.2040384Z }, 2024-06-26T07:06:46.2040636Z { 2024-06-26T07:06:46.2041057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2041590Z "size": 101, 2024-06-26T07:06:46.2042115Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:06:46.2042785Z }, 2024-06-26T07:06:46.2043029Z { 2024-06-26T07:06:46.2043454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2043998Z "size": 1473, 2024-06-26T07:06:46.2044526Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:06:46.2045112Z }, 2024-06-26T07:06:46.2045362Z { 2024-06-26T07:06:46.2045794Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2046337Z "size": 446710781, 2024-06-26T07:06:46.2046879Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:06:46.2047481Z }, 2024-06-26T07:06:46.2047737Z { 2024-06-26T07:06:46.2048163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2048700Z "size": 161, 2024-06-26T07:06:46.2049216Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:06:46.2049807Z }, 2024-06-26T07:06:46.2050051Z { 2024-06-26T07:06:46.2050476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2051020Z "size": 564, 2024-06-26T07:06:46.2051656Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:06:46.2052254Z }, 2024-06-26T07:06:46.2052503Z { 2024-06-26T07:06:46.2052929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2053474Z "size": 35874600, 2024-06-26T07:06:46.2054016Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:06:46.2054620Z }, 2024-06-26T07:06:46.2054872Z { 2024-06-26T07:06:46.2055296Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2055841Z "size": 104, 2024-06-26T07:06:46.2056360Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:06:46.2056947Z }, 2024-06-26T07:06:46.2057188Z { 2024-06-26T07:06:46.2057622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2058160Z "size": 425, 2024-06-26T07:06:46.2058677Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:06:46.2059279Z }, 2024-06-26T07:06:46.2059555Z { 2024-06-26T07:06:46.2060003Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2060544Z "size": 20262084, 2024-06-26T07:06:46.2061090Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:06:46.2061701Z }, 2024-06-26T07:06:46.2061953Z { 2024-06-26T07:06:46.2062379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2062925Z "size": 439, 2024-06-26T07:06:46.2063640Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:06:46.2064239Z }, 2024-06-26T07:06:46.2064486Z { 2024-06-26T07:06:46.2064914Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2065453Z "size": 702, 2024-06-26T07:06:46.2065989Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:46.2066597Z }, 2024-06-26T07:06:46.2066846Z { 2024-06-26T07:06:46.2067274Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2067808Z "size": 143, 2024-06-26T07:06:46.2068332Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:06:46.2068928Z }, 2024-06-26T07:06:46.2069175Z { 2024-06-26T07:06:46.2069593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2070249Z "size": 135, 2024-06-26T07:06:46.2070791Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:06:46.2071402Z }, 2024-06-26T07:06:46.2071644Z { 2024-06-26T07:06:46.2072072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2072619Z "size": 4911495247, 2024-06-26T07:06:46.2073163Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:06:46.2073757Z }, 2024-06-26T07:06:46.2074011Z { 2024-06-26T07:06:46.2074442Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2074979Z "size": 190, 2024-06-26T07:06:46.2075522Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:06:46.2076136Z }, 2024-06-26T07:06:46.2076387Z { 2024-06-26T07:06:46.2076803Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2077350Z "size": 565, 2024-06-26T07:06:46.2077891Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:06:46.2078491Z }, 2024-06-26T07:06:46.2078738Z { 2024-06-26T07:06:46.2079172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2079721Z "size": 43163095, 2024-06-26T07:06:46.2080278Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:06:46.2080887Z }, 2024-06-26T07:06:46.2081222Z { 2024-06-26T07:06:46.2081657Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2082271Z "size": 106, 2024-06-26T07:06:46.2082816Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:06:46.2083424Z }, 2024-06-26T07:06:46.2083678Z { 2024-06-26T07:06:46.2084101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2084673Z "size": 1213, 2024-06-26T07:06:46.2085210Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:06:46.2085805Z }, 2024-06-26T07:06:46.2086053Z { 2024-06-26T07:06:46.2086481Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2087024Z "size": 702, 2024-06-26T07:06:46.2087552Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:46.2088163Z }, 2024-06-26T07:06:46.2088413Z { 2024-06-26T07:06:46.2088852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2089397Z "size": 138, 2024-06-26T07:06:46.2089939Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:06:46.2090550Z }, 2024-06-26T07:06:46.2090801Z { 2024-06-26T07:06:46.2091221Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2091765Z "size": 120, 2024-06-26T07:06:46.2092312Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:06:46.2092924Z }, 2024-06-26T07:06:46.2093178Z { 2024-06-26T07:06:46.2093609Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2094157Z "size": 1904157495, 2024-06-26T07:06:46.2094714Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:06:46.2095326Z }, 2024-06-26T07:06:46.2095576Z { 2024-06-26T07:06:46.2096005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2096545Z "size": 172, 2024-06-26T07:06:46.2097086Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:06:46.2097692Z }, 2024-06-26T07:06:46.2097939Z { 2024-06-26T07:06:46.2098358Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2098900Z "size": 908, 2024-06-26T07:06:46.2099463Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:06:46.2100185Z }, 2024-06-26T07:06:46.2100431Z { 2024-06-26T07:06:46.2100855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2101403Z "size": 702, 2024-06-26T07:06:46.2101926Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:06:46.2102527Z }, 2024-06-26T07:06:46.2102778Z { 2024-06-26T07:06:46.2103198Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2103906Z "size": 134, 2024-06-26T07:06:46.2104444Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:06:46.2105039Z }, 2024-06-26T07:06:46.2105293Z { 2024-06-26T07:06:46.2105712Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2106263Z "size": 32, 2024-06-26T07:06:46.2106797Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:46.2107391Z }, 2024-06-26T07:06:46.2107648Z { 2024-06-26T07:06:46.2108074Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2108613Z "size": 156, 2024-06-26T07:06:46.2109144Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:06:46.2109754Z }, 2024-06-26T07:06:46.2110000Z { 2024-06-26T07:06:46.2110427Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2110959Z "size": 1841, 2024-06-26T07:06:46.2111591Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:06:46.2112196Z }, 2024-06-26T07:06:46.2112439Z { 2024-06-26T07:06:46.2112863Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2113410Z "size": 7529784, 2024-06-26T07:06:46.2113956Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:06:46.2114552Z }, 2024-06-26T07:06:46.2114814Z { 2024-06-26T07:06:46.2115242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2115780Z "size": 165, 2024-06-26T07:06:46.2116301Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:06:46.2116902Z }, 2024-06-26T07:06:46.2117150Z { 2024-06-26T07:06:46.2117573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2118110Z "size": 7944, 2024-06-26T07:06:46.2118641Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:06:46.2119254Z }, 2024-06-26T07:06:46.2119494Z { 2024-06-26T07:06:46.2119925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2120469Z "size": 8070, 2024-06-26T07:06:46.2121001Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:06:46.2121592Z }, 2024-06-26T07:06:46.2121841Z { 2024-06-26T07:06:46.2122361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2122911Z "size": 303, 2024-06-26T07:06:46.2123436Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:06:46.2124056Z }, 2024-06-26T07:06:46.2124309Z { 2024-06-26T07:06:46.2124723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2125266Z "size": 7629071, 2024-06-26T07:06:46.2125788Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:06:46.2126371Z }, 2024-06-26T07:06:46.2126614Z { 2024-06-26T07:06:46.2127042Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2127582Z "size": 108, 2024-06-26T07:06:46.2128113Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:06:46.2128712Z }, 2024-06-26T07:06:46.2128967Z { 2024-06-26T07:06:46.2129392Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2130066Z "size": 54145775, 2024-06-26T07:06:46.2130611Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:06:46.2131217Z }, 2024-06-26T07:06:46.2131468Z { 2024-06-26T07:06:46.2131887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2132433Z "size": 474, 2024-06-26T07:06:46.2132963Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:06:46.2133573Z }, 2024-06-26T07:06:46.2133815Z { 2024-06-26T07:06:46.2134243Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2134790Z "size": 1374858747, 2024-06-26T07:06:46.2135327Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:06:46.2135903Z }, 2024-06-26T07:06:46.2136154Z { 2024-06-26T07:06:46.2136579Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2137117Z "size": 106, 2024-06-26T07:06:46.2137654Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:06:46.2138263Z }, 2024-06-26T07:06:46.2138511Z { 2024-06-26T07:06:46.2138927Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2139468Z "size": 558, 2024-06-26T07:06:46.2140006Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:06:46.2140620Z }, 2024-06-26T07:06:46.2140926Z { 2024-06-26T07:06:46.2141355Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2141896Z "size": 46248513, 2024-06-26T07:06:46.2142435Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:06:46.2143020Z }, 2024-06-26T07:06:46.2143503Z { 2024-06-26T07:06:46.2144081Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2144626Z "size": 111, 2024-06-26T07:06:46.2145166Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:06:46.2145766Z }, 2024-06-26T07:06:46.2146014Z { 2024-06-26T07:06:46.2146433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2146976Z "size": 32, 2024-06-26T07:06:46.2147510Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:46.2148116Z }, 2024-06-26T07:06:46.2148356Z { 2024-06-26T07:06:46.2148778Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2149328Z "size": 32, 2024-06-26T07:06:46.2149861Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:46.2150459Z }, 2024-06-26T07:06:46.2150707Z { 2024-06-26T07:06:46.2151134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2151664Z "size": 32, 2024-06-26T07:06:46.2152196Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:46.2152799Z }, 2024-06-26T07:06:46.2153049Z { 2024-06-26T07:06:46.2153467Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:06:46.2154010Z "size": 32, 2024-06-26T07:06:46.2154536Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:06:46.2155139Z } 2024-06-26T07:06:46.2155379Z ] 2024-06-26T07:06:46.2155624Z } 2024-06-26T07:06:46.2271086Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:06:46.2271585Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:06:46.2272127Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:06:46.2280221Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:46.2280719Z env: 2024-06-26T07:06:46.2281005Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:46.2282138Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.2283431Z ##[endgroup] 2024-06-26T07:06:46.2305851Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.2353225Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:06:46.2353792Z with: 2024-06-26T07:06:46.2354837Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.2356132Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:46.2356647Z env: 2024-06-26T07:06:46.2356928Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:46.2357273Z ##[endgroup] 2024-06-26T07:06:46.2372497Z ##[group]Run set -x 2024-06-26T07:06:46.2372830Z set -x 2024-06-26T07:06:46.2373130Z set +e 2024-06-26T07:06:46.2373423Z  2024-06-26T07:06:46.2373694Z login() { 2024-06-26T07:06:46.2374341Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:06:46.2375048Z } 2024-06-26T07:06:46.2375321Z  2024-06-26T07:06:46.2375617Z retry () { 2024-06-26T07:06:46.2376001Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:06:46.2376435Z } 2024-06-26T07:06:46.2376703Z  2024-06-26T07:06:46.2377015Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:06:46.2377522Z  2024-06-26T07:06:46.2377787Z set -e 2024-06-26T07:06:46.2378253Z # ignore output since only exit code is used for conditional 2024-06-26T07:06:46.2378936Z # only pull docker image if it's not available locally 2024-06-26T07:06:46.2379721Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:06:46.2380403Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:06:46.2380827Z fi 2024-06-26T07:06:46.2387730Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:06:46.2388218Z env: 2024-06-26T07:06:46.2388494Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:06:46.2389614Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.2390894Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:46.2391410Z ##[endgroup] 2024-06-26T07:06:46.2410916Z + set +e 2024-06-26T07:06:46.2411602Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:46.2412231Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:46.2413101Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:06:46.2413970Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:06:46.6577152Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:06:46.6578199Z Configure a credential helper to remove this warning. See 2024-06-26T07:06:46.6579073Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:06:46.6579616Z 2024-06-26T07:06:46.6579739Z Login Succeeded 2024-06-26T07:06:46.6587293Z + set -e 2024-06-26T07:06:46.6588599Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.6715403Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.6717496Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:06:46.8669838Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:06:46.8683873Z 7a2c55901189: Pulling fs layer 2024-06-26T07:06:46.8684459Z 224fe954d725: Pulling fs layer 2024-06-26T07:06:46.8685257Z 75722010b82e: Pulling fs layer 2024-06-26T07:06:46.8685824Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:06:46.8686410Z b57676e46aee: Pulling fs layer 2024-06-26T07:06:46.8686971Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:06:46.8687675Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:06:46.8688222Z 0c1227890755: Pulling fs layer 2024-06-26T07:06:46.8688757Z d8d1234baab3: Pulling fs layer 2024-06-26T07:06:46.8689317Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:06:46.8689883Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:06:46.8690492Z 309b872da805: Pulling fs layer 2024-06-26T07:06:46.8691028Z e82932cb002c: Pulling fs layer 2024-06-26T07:06:46.8691411Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:06:46.8691788Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:06:46.8692193Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:06:46.8692708Z f4863801e30c: Pulling fs layer 2024-06-26T07:06:46.8693198Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:06:46.8693674Z d527cbbb87e3: Waiting 2024-06-26T07:06:46.8694119Z cf2f34f225db: Pulling fs layer 2024-06-26T07:06:46.8694543Z df546d7b3e34: Pulling fs layer 2024-06-26T07:06:46.8694910Z b57676e46aee: Waiting 2024-06-26T07:06:46.8695238Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:06:46.8695718Z 96d0e2032184: Pulling fs layer 2024-06-26T07:06:46.8696364Z a8c1e85b5e14: Waiting 2024-06-26T07:06:46.8696803Z 06e12ce59790: Pulling fs layer 2024-06-26T07:06:46.8697264Z a41a8d1c11c8: Waiting 2024-06-26T07:06:46.8697704Z ed712f63cde7: Pulling fs layer 2024-06-26T07:06:46.8698212Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:06:46.8698757Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:06:46.8699265Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:06:46.8699655Z 0c1227890755: Waiting 2024-06-26T07:06:46.8699976Z d37dd9ed8948: Waiting 2024-06-26T07:06:46.8700295Z 16d76a79d5d6: Waiting 2024-06-26T07:06:46.8700619Z de45a14c9b22: Pulling fs layer 2024-06-26T07:06:46.8700983Z cf2f34f225db: Waiting 2024-06-26T07:06:46.8701313Z e99241e77209: Pulling fs layer 2024-06-26T07:06:46.8701698Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:06:46.8702063Z a823ffa4c541: Pulling fs layer 2024-06-26T07:06:46.8702537Z 44ba272eae20: Pulling fs layer 2024-06-26T07:06:46.8702912Z da02d278e524: Pulling fs layer 2024-06-26T07:06:46.8703647Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:06:46.8704094Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:06:46.8704450Z 96d0e2032184: Waiting 2024-06-26T07:06:46.8704781Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:06:46.8705145Z b2950a036534: Pulling fs layer 2024-06-26T07:06:46.8705507Z 4bdf41c9ecbe: Waiting 2024-06-26T07:06:46.8705825Z 309b872da805: Waiting 2024-06-26T07:06:46.8706154Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:06:46.8706517Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:06:46.8706885Z 7a868239227f: Pulling fs layer 2024-06-26T07:06:46.8707238Z f4863801e30c: Waiting 2024-06-26T07:06:46.8707550Z 3d914950f8ad: Waiting 2024-06-26T07:06:46.8707872Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:06:46.8708227Z e82932cb002c: Waiting 2024-06-26T07:06:46.8708560Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:06:46.8708913Z 44ba272eae20: Waiting 2024-06-26T07:06:46.8709230Z d8d1234baab3: Waiting 2024-06-26T07:06:46.8709544Z 06e12ce59790: Waiting 2024-06-26T07:06:46.8709908Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:06:46.8710289Z c1068d03a811: Pulling fs layer 2024-06-26T07:06:46.8710643Z da02d278e524: Waiting 2024-06-26T07:06:46.8710960Z ed712f63cde7: Waiting 2024-06-26T07:06:46.8711268Z df546d7b3e34: Waiting 2024-06-26T07:06:46.8711585Z 35e8f4e2a42f: Waiting 2024-06-26T07:06:46.8711910Z 671b4a301d81: Pulling fs layer 2024-06-26T07:06:46.8712279Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:06:46.8712624Z de45a14c9b22: Waiting 2024-06-26T07:06:46.8712960Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:06:46.8713406Z fb2e97768560: Pulling fs layer 2024-06-26T07:06:46.8713768Z e99241e77209: Waiting 2024-06-26T07:06:46.8714118Z ca9dd73e3e93: Waiting 2024-06-26T07:06:46.8714452Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:06:46.8714953Z e08e88c13668: Pulling fs layer 2024-06-26T07:06:46.8715483Z 9bf94bc45658: Waiting 2024-06-26T07:06:46.8715818Z 9cd4c745f8e4: Waiting 2024-06-26T07:06:46.8716134Z 49737bd5ae70: Waiting 2024-06-26T07:06:46.8716462Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:06:46.8716813Z ec1e7978c1fe: Waiting 2024-06-26T07:06:46.8717148Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:06:46.8717528Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:06:46.8717886Z c1068d03a811: Waiting 2024-06-26T07:06:46.8718192Z b599c5a8c7cb: Waiting 2024-06-26T07:06:46.8718506Z a823ffa4c541: Waiting 2024-06-26T07:06:46.8718819Z 671b4a301d81: Waiting 2024-06-26T07:06:46.8719127Z 83911d7c7ecf: Waiting 2024-06-26T07:06:46.8719436Z 25e19d81d9fc: Waiting 2024-06-26T07:06:46.8719750Z b3c5d66c679f: Waiting 2024-06-26T07:06:46.8720059Z 60156b40fcbe: Waiting 2024-06-26T07:06:46.8720370Z 84b2c7a89982: Waiting 2024-06-26T07:06:46.8720687Z ddc9740bcea2: Waiting 2024-06-26T07:06:46.8721001Z 3a778ddfac2a: Waiting 2024-06-26T07:06:46.8721308Z cf7c2cb896cb: Waiting 2024-06-26T07:06:46.8721625Z 12ba7431ff38: Waiting 2024-06-26T07:06:46.8721937Z fb2e97768560: Waiting 2024-06-26T07:06:46.8722363Z 7a868239227f: Waiting 2024-06-26T07:06:46.8722686Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:06:46.8723065Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:06:46.8723528Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:06:46.8723880Z 236c8f0245f4: Waiting 2024-06-26T07:06:46.8724208Z e797fe86caef: Pulling fs layer 2024-06-26T07:06:46.8724582Z 967190076fc0: Pulling fs layer 2024-06-26T07:06:46.8724950Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:06:46.8725312Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:06:46.8725686Z cc91933e985a: Pulling fs layer 2024-06-26T07:06:46.8726058Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:06:46.8726417Z e797fe86caef: Waiting 2024-06-26T07:06:46.8726737Z 526083994857: Pulling fs layer 2024-06-26T07:06:46.8727106Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:06:46.8727475Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:06:46.8727827Z 4fb0a83f0685: Waiting 2024-06-26T07:06:46.8728225Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:06:46.8728596Z 95237e7311a1: Pulling fs layer 2024-06-26T07:06:46.8728950Z cc91933e985a: Waiting 2024-06-26T07:06:46.8729258Z 68af2ec4e30a: Waiting 2024-06-26T07:06:46.8729590Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:06:46.8729967Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:06:46.8730345Z 2060028f7488: Pulling fs layer 2024-06-26T07:06:46.8730688Z 9f3187916e7d: Waiting 2024-06-26T07:06:46.8731020Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:06:46.8731371Z 526083994857: Waiting 2024-06-26T07:06:46.8731676Z 05a6f6dcc1c0: Waiting 2024-06-26T07:06:46.8731991Z 789eff6c51aa: Waiting 2024-06-26T07:06:46.8732308Z e412c7f3b2de: Waiting 2024-06-26T07:06:46.8732623Z 6bc9dc8315b0: Waiting 2024-06-26T07:06:47.0228642Z 224fe954d725: Verifying Checksum 2024-06-26T07:06:47.0229065Z 224fe954d725: Download complete 2024-06-26T07:06:47.1089821Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:06:47.1090231Z d527cbbb87e3: Download complete 2024-06-26T07:06:47.1950616Z b57676e46aee: Verifying Checksum 2024-06-26T07:06:47.1951039Z b57676e46aee: Download complete 2024-06-26T07:06:47.2035740Z 7a2c55901189: Verifying Checksum 2024-06-26T07:06:47.2036142Z 7a2c55901189: Download complete 2024-06-26T07:06:47.2872596Z a41a8d1c11c8: Verifying Checksum 2024-06-26T07:06:47.2873178Z a41a8d1c11c8: Download complete 2024-06-26T07:06:47.3700644Z 0c1227890755: Verifying Checksum 2024-06-26T07:06:47.3701103Z 0c1227890755: Download complete 2024-06-26T07:06:47.4319223Z d8d1234baab3: Verifying Checksum 2024-06-26T07:06:47.4319694Z d8d1234baab3: Download complete 2024-06-26T07:06:47.4789725Z 75722010b82e: Verifying Checksum 2024-06-26T07:06:47.4790193Z 75722010b82e: Download complete 2024-06-26T07:06:47.5920582Z ec1e7978c1fe: Download complete 2024-06-26T07:06:47.6560871Z 309b872da805: Verifying Checksum 2024-06-26T07:06:47.6561498Z 309b872da805: Download complete 2024-06-26T07:06:48.2382030Z 7a2c55901189: Pull complete 2024-06-26T07:06:48.5271757Z 224fe954d725: Pull complete 2024-06-26T07:06:49.4775091Z 75722010b82e: Pull complete 2024-06-26T07:06:49.5277864Z d527cbbb87e3: Pull complete 2024-06-26T07:06:49.5882275Z b57676e46aee: Pull complete 2024-06-26T07:06:50.1706903Z e82932cb002c: Verifying Checksum 2024-06-26T07:06:50.1707394Z e82932cb002c: Download complete 2024-06-26T07:06:50.2471032Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:06:50.2471646Z ca9dd73e3e93: Download complete 2024-06-26T07:06:50.3321373Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:06:50.3322007Z 16d76a79d5d6: Download complete 2024-06-26T07:06:50.4103106Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:06:50.4104193Z 4bdf41c9ecbe: Download complete 2024-06-26T07:06:51.3878294Z f4863801e30c: Verifying Checksum 2024-06-26T07:06:51.3878782Z f4863801e30c: Download complete 2024-06-26T07:06:51.4579845Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:06:51.4580324Z d37dd9ed8948: Download complete 2024-06-26T07:06:51.5430894Z cf2f34f225db: Download complete 2024-06-26T07:06:51.6171926Z df546d7b3e34: Verifying Checksum 2024-06-26T07:06:51.6172539Z df546d7b3e34: Download complete 2024-06-26T07:07:00.5401763Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:07:00.5402347Z a8c1e85b5e14: Download complete 2024-06-26T07:07:00.6199650Z 96d0e2032184: Verifying Checksum 2024-06-26T07:07:00.7049299Z 96d0e2032184: Download complete 2024-06-26T07:07:00.7050187Z 06e12ce59790: Verifying Checksum 2024-06-26T07:07:00.7050924Z 06e12ce59790: Download complete 2024-06-26T07:07:00.7812854Z ed712f63cde7: Verifying Checksum 2024-06-26T07:07:00.7813537Z ed712f63cde7: Download complete 2024-06-26T07:07:00.8646052Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:07:00.8646788Z a97a71ef8a4e: Download complete 2024-06-26T07:07:00.9433801Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:07:00.9434483Z 25cd0c482ad6: Download complete 2024-06-26T07:07:02.2428452Z 9bf94bc45658: Download complete 2024-06-26T07:07:02.3254673Z de45a14c9b22: Download complete 2024-06-26T07:07:02.4029882Z e99241e77209: Download complete 2024-06-26T07:07:02.4909399Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:07:02.4909897Z 35e8f4e2a42f: Download complete 2024-06-26T07:07:02.5639600Z a823ffa4c541: Download complete 2024-06-26T07:07:02.6457550Z 44ba272eae20: Download complete 2024-06-26T07:07:07.1774242Z da02d278e524: Verifying Checksum 2024-06-26T07:07:07.1774800Z da02d278e524: Download complete 2024-06-26T07:07:07.2399130Z 73a0ce681c29: Download complete 2024-06-26T07:07:07.3088514Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:07:07.3089169Z 49737bd5ae70: Download complete 2024-06-26T07:07:07.7286950Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:07:07.7287449Z 9cd4c745f8e4: Download complete 2024-06-26T07:07:07.8081140Z b2950a036534: Verifying Checksum 2024-06-26T07:07:07.8081713Z b2950a036534: Download complete 2024-06-26T07:07:07.8758306Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:07:07.8758991Z 25e19d81d9fc: Download complete 2024-06-26T07:07:08.1455747Z 60156b40fcbe: Download complete 2024-06-26T07:07:08.2390997Z 7a868239227f: Verifying Checksum 2024-06-26T07:07:08.2391457Z 7a868239227f: Download complete 2024-06-26T07:07:08.3319740Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:07:08.3320356Z 83911d7c7ecf: Download complete 2024-06-26T07:07:08.4000840Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:07:08.4001364Z b3c5d66c679f: Download complete 2024-06-26T07:07:12.9818261Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:07:12.9818877Z 7ed32bc8e469: Download complete 2024-06-26T07:07:13.0538001Z c1068d03a811: Verifying Checksum 2024-06-26T07:07:13.0538436Z c1068d03a811: Download complete 2024-06-26T07:07:13.1445662Z 671b4a301d81: Verifying Checksum 2024-06-26T07:07:13.1446292Z 671b4a301d81: Download complete 2024-06-26T07:07:13.2864995Z a8c1e85b5e14: Pull complete 2024-06-26T07:07:13.5190890Z a41a8d1c11c8: Pull complete 2024-06-26T07:07:13.6212904Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:07:13.6213340Z 67760cf5ec46: Download complete 2024-06-26T07:07:13.6950995Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:07:13.6951423Z ddc9740bcea2: Download complete 2024-06-26T07:07:13.7397989Z 0c1227890755: Pull complete 2024-06-26T07:07:13.7667384Z fb2e97768560: Verifying Checksum 2024-06-26T07:07:13.7667803Z fb2e97768560: Download complete 2024-06-26T07:07:13.8328150Z b599c5a8c7cb: Download complete 2024-06-26T07:07:13.9025973Z e08e88c13668: Download complete 2024-06-26T07:07:13.9421450Z d8d1234baab3: Pull complete 2024-06-26T07:07:19.5165455Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:07:19.5165921Z 3d914950f8ad: Download complete 2024-06-26T07:07:19.5910045Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:07:19.5910511Z cf7c2cb896cb: Download complete 2024-06-26T07:07:19.6735275Z 3a778ddfac2a: Download complete 2024-06-26T07:07:19.7409797Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:07:19.7410444Z 236c8f0245f4: Download complete 2024-06-26T07:07:19.7529308Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:07:19.7529818Z 4f4fb700ef54: Download complete 2024-06-26T07:07:19.8174286Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:07:19.8174746Z dda1ce302b9a: Download complete 2024-06-26T07:07:19.8969554Z e797fe86caef: Download complete 2024-06-26T07:07:20.0291075Z 967190076fc0: Verifying Checksum 2024-06-26T07:07:20.0291606Z 967190076fc0: Download complete 2024-06-26T07:07:20.1295120Z 4fb0a83f0685: Download complete 2024-06-26T07:07:20.2034406Z 68af2ec4e30a: Download complete 2024-06-26T07:07:20.2828595Z cc91933e985a: Download complete 2024-06-26T07:07:20.3495907Z 9f3187916e7d: Download complete 2024-06-26T07:07:20.4829243Z 526083994857: Verifying Checksum 2024-06-26T07:07:20.4829712Z 526083994857: Download complete 2024-06-26T07:07:20.5591205Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:07:20.5591629Z 05a6f6dcc1c0: Download complete 2024-06-26T07:07:21.1643948Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:07:21.1644429Z 6bc9dc8315b0: Download complete 2024-06-26T07:07:21.2526682Z 1b0e27c35f07: Download complete 2024-06-26T07:07:44.6815661Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:07:44.6816130Z 12ba7431ff38: Download complete 2024-06-26T07:07:44.7648421Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:07:44.7648965Z 789eff6c51aa: Download complete 2024-06-26T07:07:44.8856738Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:07:44.8857306Z de0f8a8c45c6: Download complete 2024-06-26T07:07:46.1839529Z 2060028f7488: Verifying Checksum 2024-06-26T07:07:46.1840021Z 2060028f7488: Download complete 2024-06-26T07:07:46.2700190Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:07:46.2701015Z e412c7f3b2de: Download complete 2024-06-26T07:07:48.9434416Z 95237e7311a1: Verifying Checksum 2024-06-26T07:07:48.9434959Z 95237e7311a1: Download complete 2024-06-26T07:08:12.8424931Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:08:29.5627548Z 7ed32bc8e469: Pull complete 2024-06-26T07:08:29.7849880Z ec1e7978c1fe: Pull complete 2024-06-26T07:08:29.9793862Z 309b872da805: Pull complete 2024-06-26T07:08:37.3202061Z e82932cb002c: Pull complete 2024-06-26T07:08:37.5346956Z ca9dd73e3e93: Pull complete 2024-06-26T07:08:37.7393341Z 16d76a79d5d6: Pull complete 2024-06-26T07:08:37.9440436Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:08:40.3576512Z f4863801e30c: Pull complete 2024-06-26T07:08:40.5722161Z d37dd9ed8948: Pull complete 2024-06-26T07:08:40.7789755Z cf2f34f225db: Pull complete 2024-06-26T07:08:40.9774388Z df546d7b3e34: Pull complete 2024-06-26T07:09:30.3148642Z 3d914950f8ad: Pull complete 2024-06-26T07:09:30.5087641Z 96d0e2032184: Pull complete 2024-06-26T07:09:30.6446274Z 06e12ce59790: Pull complete 2024-06-26T07:09:30.7985152Z ed712f63cde7: Pull complete 2024-06-26T07:09:30.9120522Z a97a71ef8a4e: Pull complete 2024-06-26T07:09:31.0098498Z 25cd0c482ad6: Pull complete 2024-06-26T07:09:34.5187686Z 9bf94bc45658: Pull complete 2024-06-26T07:09:34.7312551Z de45a14c9b22: Pull complete 2024-06-26T07:09:34.9362042Z e99241e77209: Pull complete 2024-06-26T07:09:35.1501157Z 35e8f4e2a42f: Pull complete 2024-06-26T07:09:35.3565585Z a823ffa4c541: Pull complete 2024-06-26T07:09:35.5655989Z 44ba272eae20: Pull complete 2024-06-26T07:09:43.8790348Z da02d278e524: Pull complete 2024-06-26T07:09:44.0253921Z 73a0ce681c29: Pull complete 2024-06-26T07:09:44.1999608Z 49737bd5ae70: Pull complete 2024-06-26T07:09:45.1483935Z 9cd4c745f8e4: Pull complete 2024-06-26T07:09:45.2341553Z b2950a036534: Pull complete 2024-06-26T07:09:45.4419608Z 25e19d81d9fc: Pull complete 2024-06-26T07:09:45.9333971Z 60156b40fcbe: Pull complete 2024-06-26T07:09:46.0914501Z 7a868239227f: Pull complete 2024-06-26T07:09:46.4255400Z 83911d7c7ecf: Pull complete 2024-06-26T07:09:46.6031911Z b3c5d66c679f: Pull complete 2024-06-26T07:10:25.5847794Z 84b2c7a89982: Pull complete 2024-06-26T07:10:25.7285896Z c1068d03a811: Pull complete 2024-06-26T07:10:25.7682691Z 671b4a301d81: Pull complete 2024-06-26T07:10:27.7233496Z 67760cf5ec46: Pull complete 2024-06-26T07:10:27.7541384Z ddc9740bcea2: Pull complete 2024-06-26T07:10:27.8067024Z fb2e97768560: Pull complete 2024-06-26T07:10:27.9064145Z b599c5a8c7cb: Pull complete 2024-06-26T07:10:27.9515058Z e08e88c13668: Pull complete 2024-06-26T07:11:02.0890443Z 12ba7431ff38: Pull complete 2024-06-26T07:11:02.2920888Z cf7c2cb896cb: Pull complete 2024-06-26T07:11:02.4646521Z 3a778ddfac2a: Pull complete 2024-06-26T07:11:02.8961022Z 236c8f0245f4: Pull complete 2024-06-26T07:11:03.0214232Z 4f4fb700ef54: Pull complete 2024-06-26T07:11:03.1382009Z dda1ce302b9a: Pull complete 2024-06-26T07:11:03.3596937Z e797fe86caef: Pull complete 2024-06-26T07:11:03.7441790Z 967190076fc0: Pull complete 2024-06-26T07:11:03.9353872Z 4fb0a83f0685: Pull complete 2024-06-26T07:11:04.1253875Z 68af2ec4e30a: Pull complete 2024-06-26T07:11:04.3248545Z cc91933e985a: Pull complete 2024-06-26T07:11:04.5222602Z 9f3187916e7d: Pull complete 2024-06-26T07:11:05.6224727Z 526083994857: Pull complete 2024-06-26T07:11:05.7961896Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:11:07.8467977Z 6bc9dc8315b0: Pull complete 2024-06-26T07:11:08.0397665Z 1b0e27c35f07: Pull complete 2024-06-26T07:11:21.6511415Z 95237e7311a1: Pull complete 2024-06-26T07:11:21.6976540Z 789eff6c51aa: Pull complete 2024-06-26T07:11:21.7390710Z de0f8a8c45c6: Pull complete 2024-06-26T07:11:22.5264488Z 2060028f7488: Pull complete 2024-06-26T07:11:22.7377333Z e412c7f3b2de: Pull complete 2024-06-26T07:11:23.8340510Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:11:23.8782315Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:23.8984255Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:23.9035608Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:23.9036526Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:11:23.9044771Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:23.9045266Z env: 2024-06-26T07:11:23.9045545Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:23.9045894Z ##[endgroup] 2024-06-26T07:11:23.9199542Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:11:23.9200082Z with: 2024-06-26T07:11:23.9200373Z driver-version: 550.54.15 2024-06-26T07:11:23.9200716Z env: 2024-06-26T07:11:23.9200999Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:23.9201345Z ##[endgroup] 2024-06-26T07:11:23.9320745Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:11:23.9321287Z with: 2024-06-26T07:11:23.9321560Z timeout_minutes: 10 2024-06-26T07:11:23.9321876Z max_attempts: 3 2024-06-26T07:11:23.9352812Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:11:23.9383965Z retry_wait_seconds: 10 2024-06-26T07:11:23.9384328Z polling_interval_seconds: 1 2024-06-26T07:11:23.9384696Z warning_on_retry: true 2024-06-26T07:11:23.9385044Z continue_on_error: false 2024-06-26T07:11:23.9385378Z env: 2024-06-26T07:11:23.9385648Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:23.9385997Z DRIVER_VERSION: 550.54.15 2024-06-26T07:11:23.9386337Z ##[endgroup] 2024-06-26T07:11:23.9906541Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:11:23.9907336Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:11:23.9908073Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:11:24.3066223Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:24.3466792Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:11:24.3751864Z No Packages marked for removal 2024-06-26T07:11:24.3873446Z + install_nvidia_driver_common 2024-06-26T07:11:24.3875626Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:11:24.3876082Z Before installing NVIDIA driver 2024-06-26T07:11:24.3877961Z + lspci 2024-06-26T07:11:24.4070406Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:24.4071168Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:24.4072150Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:24.4072935Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:24.4073674Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:24.4074482Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:24.4075195Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:24.4075973Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:24.4076552Z + lsmod 2024-06-26T07:11:24.4237427Z Module Size Used by 2024-06-26T07:11:24.4237851Z xt_nat 16384 0 2024-06-26T07:11:24.4238251Z nvidia_modeset 1339392 0 2024-06-26T07:11:24.4238630Z veth 16384 0 2024-06-26T07:11:24.4238993Z nvidia_uvm 4599808 0 2024-06-26T07:11:24.4239443Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:24.4240081Z drm 421888 1 nvidia 2024-06-26T07:11:24.4241599Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:24.4242414Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:24.4243036Z xt_conntrack 16384 1 2024-06-26T07:11:24.4243511Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:24.4243953Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:24.4244420Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:24.4245010Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:24.4245643Z xfrm_user 45056 1 2024-06-26T07:11:24.4246139Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:24.4246706Z iptable_nat 16384 1 2024-06-26T07:11:24.4247287Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:24.4247847Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:24.4248417Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:24.4249148Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:11:24.4250394Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:11:24.4251257Z xt_addrtype 16384 2 2024-06-26T07:11:24.4251648Z iptable_filter 16384 1 2024-06-26T07:11:24.4252060Z br_netfilter 24576 0 2024-06-26T07:11:24.4252479Z bridge 172032 1 br_netfilter 2024-06-26T07:11:24.4253041Z stp 16384 1 bridge 2024-06-26T07:11:24.4253606Z llc 16384 2 bridge,stp 2024-06-26T07:11:24.4254141Z overlay 86016 0 2024-06-26T07:11:24.4254513Z sunrpc 393216 1 2024-06-26T07:11:24.4254875Z dm_mirror 28672 0 2024-06-26T07:11:24.4255265Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:24.4255749Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:24.4256264Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:24.4256801Z dax 69632 1 dm_mod 2024-06-26T07:11:24.4257199Z crc32_pclmul 16384 0 2024-06-26T07:11:24.4257615Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:24.4257987Z pcbc 16384 0 2024-06-26T07:11:24.4258588Z aesni_intel 188416 0 2024-06-26T07:11:24.4258981Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:24.4259431Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:24.4259873Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:24.4260293Z mousedev 24576 0 2024-06-26T07:11:24.4260786Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:24.4261320Z psmouse 32768 0 2024-06-26T07:11:24.4261677Z evdev 20480 3 2024-06-26T07:11:24.4262038Z button 16384 0 2024-06-26T07:11:24.4262402Z ena 139264 0 2024-06-26T07:11:24.4262773Z ptp 20480 1 ena 2024-06-26T07:11:24.4263172Z pps_core 20480 1 ptp 2024-06-26T07:11:24.4263869Z crc32c_intel 24576 0 2024-06-26T07:11:24.4264243Z autofs4 49152 2 2024-06-26T07:11:24.4264605Z + modinfo nvidia 2024-06-26T07:11:24.4265404Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:24.4266104Z alias: char-major-195-* 2024-06-26T07:11:24.4266491Z version: 550.54.15 2024-06-26T07:11:24.4266838Z supported: external 2024-06-26T07:11:24.4267180Z license: NVIDIA 2024-06-26T07:11:24.4267569Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:24.4268051Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:24.4268506Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:24.4268978Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:24.4269479Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:24.4269959Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:24.4270569Z depends: i2c-core,drm 2024-06-26T07:11:24.4270939Z retpoline: Y 2024-06-26T07:11:24.4271252Z name: nvidia 2024-06-26T07:11:24.4271838Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:24.4272503Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:24.4273149Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:24.4273746Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:24.4274192Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:24.4274624Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:24.4275087Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:24.4275526Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:24.4275969Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:24.4276492Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:24.4277066Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:24.4277551Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:24.4277979Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:24.4278408Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:24.4278869Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:24.4279391Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:24.4279962Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:24.4280524Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:24.4281131Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:24.4281727Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:24.4282437Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:24.4283026Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:24.4283524Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:24.4284062Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:24.4284602Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:24.4285098Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:24.4285561Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:24.4286147Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:24.4286611Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:24.4287053Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:24.4287551Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:24.4288073Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:24.4288549Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:24.4289031Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:24.4289523Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:24.4289993Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:24.4290489Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:24.4290964Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:24.4291374Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:24.4291833Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:24.4292301Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:24.4292767Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:24.4293238Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:24.4293702Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:24.4294154Z parm: rm_firmware_active:charp 2024-06-26T07:11:24.4294563Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:11:24.4294949Z ++ command -v nvidia-smi 2024-06-26T07:11:24.4295331Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:11:24.4295696Z + set +e 2024-06-26T07:11:24.4296196Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:11:26.4572230Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:11:26.4572867Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:26.4573476Z + '[' 0 -ne 0 ']' 2024-06-26T07:11:26.4574193Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:11:26.4574750Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:11:26.4575588Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:11:26.4576301Z + set -e 2024-06-26T07:11:26.4576599Z + '[' 1 -eq 0 ']' 2024-06-26T07:11:26.4577376Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:11:26.4578049Z + post_install_nvidia_driver_common 2024-06-26T07:11:26.4578458Z + sudo modprobe nvidia 2024-06-26T07:11:26.4672328Z + echo 'After installing NVIDIA driver' 2024-06-26T07:11:26.4672841Z + lspci 2024-06-26T07:11:26.4673494Z After installing NVIDIA driver 2024-06-26T07:11:26.4749995Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:11:26.4750925Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:11:26.4751960Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:11:26.4752748Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:11:26.4753482Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:11:26.4754218Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:11:26.4754941Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:11:26.4755768Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:11:26.4756410Z + lsmod 2024-06-26T07:11:26.4764194Z Module Size Used by 2024-06-26T07:11:26.4764794Z xt_nat 16384 0 2024-06-26T07:11:26.4765296Z nvidia_modeset 1339392 0 2024-06-26T07:11:26.4765767Z veth 16384 0 2024-06-26T07:11:26.4766140Z nvidia_uvm 4599808 0 2024-06-26T07:11:26.4766587Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:11:26.4767101Z drm 421888 1 nvidia 2024-06-26T07:11:26.4767523Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:11:26.4767987Z backlight 16384 1 nvidia_modeset 2024-06-26T07:11:26.4768426Z xt_conntrack 16384 1 2024-06-26T07:11:26.4768803Z ipt_MASQUERADE 16384 1 2024-06-26T07:11:26.4769414Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:11:26.4769885Z nf_conntrack_netlink 49152 0 2024-06-26T07:11:26.4770331Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:11:26.4770786Z xfrm_user 45056 1 2024-06-26T07:11:26.4771170Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:11:26.4771636Z iptable_nat 16384 1 2024-06-26T07:11:26.4772166Z nf_conntrack_ipv4 16384 3 2024-06-26T07:11:26.4772754Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:11:26.4773226Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:11:26.4773764Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:11:26.4774772Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:11:26.4775627Z xt_addrtype 16384 2 2024-06-26T07:11:26.4775995Z iptable_filter 16384 1 2024-06-26T07:11:26.4776379Z br_netfilter 24576 0 2024-06-26T07:11:26.4776781Z bridge 172032 1 br_netfilter 2024-06-26T07:11:26.4777222Z stp 16384 1 bridge 2024-06-26T07:11:26.4777631Z llc 16384 2 bridge,stp 2024-06-26T07:11:26.4778040Z overlay 86016 0 2024-06-26T07:11:26.4778401Z sunrpc 393216 1 2024-06-26T07:11:26.4778761Z dm_mirror 28672 0 2024-06-26T07:11:26.4779149Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:11:26.4779663Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:11:26.4780164Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:11:26.4780694Z dax 69632 1 dm_mod 2024-06-26T07:11:26.4781239Z crc32_pclmul 16384 0 2024-06-26T07:11:26.4781616Z ghash_clmulni_intel 16384 0 2024-06-26T07:11:26.4781991Z pcbc 16384 0 2024-06-26T07:11:26.4782344Z aesni_intel 188416 0 2024-06-26T07:11:26.4782738Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:11:26.4783184Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:11:26.4783905Z glue_helper 16384 1 aesni_intel 2024-06-26T07:11:26.4784322Z mousedev 24576 0 2024-06-26T07:11:26.4784820Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:11:26.4785354Z psmouse 32768 0 2024-06-26T07:11:26.4785779Z evdev 20480 3 2024-06-26T07:11:26.4786141Z button 16384 0 2024-06-26T07:11:26.4786501Z ena 139264 0 2024-06-26T07:11:26.4786868Z ptp 20480 1 ena 2024-06-26T07:11:26.4787264Z pps_core 20480 1 ptp 2024-06-26T07:11:26.4787654Z crc32c_intel 24576 0 2024-06-26T07:11:26.4788019Z autofs4 49152 2 2024-06-26T07:11:26.4788375Z + modinfo nvidia 2024-06-26T07:11:26.4789024Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:11:26.4789707Z alias: char-major-195-* 2024-06-26T07:11:26.4790085Z version: 550.54.15 2024-06-26T07:11:26.4790430Z supported: external 2024-06-26T07:11:26.4790772Z license: NVIDIA 2024-06-26T07:11:26.4791140Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:11:26.4791623Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:11:26.4792079Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:11:26.4792583Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:11:26.4793077Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:11:26.4793565Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:11:26.4794050Z depends: i2c-core,drm 2024-06-26T07:11:26.4794407Z retpoline: Y 2024-06-26T07:11:26.4794714Z name: nvidia 2024-06-26T07:11:26.4795257Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:11:26.4795900Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:11:26.4796620Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:11:26.4797207Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:11:26.4797648Z parm: NVreg_RmLogonRC:int 2024-06-26T07:11:26.4798080Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:11:26.4798520Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:11:26.4798950Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:11:26.4799386Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:11:26.4799910Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:11:26.4800458Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:11:26.4800929Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:11:26.4801367Z parm: NVreg_EnableMSI:int 2024-06-26T07:11:26.4801772Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:11:26.4802360Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:11:26.4802886Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:11:26.4803462Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:11:26.4804000Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:11:26.4804596Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:26.4805178Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:11:26.4805774Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:11:26.4806345Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:11:26.4806822Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:11:26.4807342Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:11:26.4807958Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:11:26.4808448Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:11:26.4808903Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:11:26.4809374Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:11:26.4809840Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:11:26.4810270Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:11:26.4810763Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:11:26.4811279Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:11:26.4811751Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:11:26.4812220Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:11:26.4812707Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:11:26.4813185Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:11:26.4813666Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:11:26.4814134Z parm: NVreg_RmMsg:charp 2024-06-26T07:11:26.4814548Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:11:26.4815002Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:11:26.4815459Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:11:26.4815906Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:11:26.4816376Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:11:26.4816840Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:11:26.4817274Z parm: rm_firmware_active:charp 2024-06-26T07:11:26.4817663Z + set +e 2024-06-26T07:11:26.4817965Z + nvidia-smi 2024-06-26T07:11:28.0498149Z Wed Jun 26 07:11:28 2024 2024-06-26T07:11:28.0499013Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:28.0499874Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:11:28.0500654Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:28.0501474Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:11:28.0502357Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:11:28.0503096Z | | | MIG M. | 2024-06-26T07:11:28.0504039Z |=========================================+========================+======================| 2024-06-26T07:11:28.0629374Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:11:28.0630103Z | 0% 32C P0 59W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:11:28.0630729Z | | | N/A | 2024-06-26T07:11:28.0631393Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:11:28.0632082Z 2024-06-26T07:11:28.0632742Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:28.0633357Z | Processes: | 2024-06-26T07:11:28.0634057Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:11:28.0634737Z | ID ID Usage | 2024-06-26T07:11:28.0635298Z |=========================================================================================| 2024-06-26T07:11:28.0635939Z | No running processes found | 2024-06-26T07:11:28.0636677Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:11:28.6932297Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:11:30.2529577Z NVIDIA A10G 2024-06-26T07:11:30.6883640Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:11:30.6884343Z + '[' 0 -eq 0 ']' 2024-06-26T07:11:30.6884834Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:11:30.6885339Z + set -e 2024-06-26T07:11:30.6885649Z INFO: Ignoring allowed status 0 2024-06-26T07:11:30.6886666Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:11:30.6889035Z + sudo yum install -y yum-utils 2024-06-26T07:11:30.9927345Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:32.5458255Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:11:32.5458944Z Nothing to do 2024-06-26T07:11:32.6945584Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:11:32.6946427Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:32.6947460Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:33.0169556Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:33.0436505Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:11:33.0437660Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:33.0438619Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:11:33.0556691Z + sudo yum install -y nvidia-docker2 2024-06-26T07:11:33.3674706Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:11:34.8427035Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:11:34.8427715Z Nothing to do 2024-06-26T07:11:34.9890251Z + sudo systemctl restart docker 2024-06-26T07:12:11.0341733Z Command completed after 1 attempt(s). 2024-06-26T07:12:11.0398233Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:11.0399006Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:12:11.0399692Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:12:11.0400355Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:11.0408422Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:11.0409069Z env: 2024-06-26T07:12:11.0409365Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:11.0409831Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:11.0410319Z ##[endgroup] 2024-06-26T07:12:11.2680824Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:11.2837911Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:12:11.2937111Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:12:11.3874252Z Prepare all required actions 2024-06-26T07:12:11.3874736Z Getting action download info 2024-06-26T07:12:11.4898893Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:12:11.7627178Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:12:12.9063099Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:12:12.9064010Z with: 2024-06-26T07:12:12.9064506Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:12.9065180Z s3-bucket: gha-artifacts 2024-06-26T07:12:12.9065677Z env: 2024-06-26T07:12:12.9066074Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:12.9066737Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:12.9067475Z ##[endgroup] 2024-06-26T07:12:12.9114239Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:12.9114897Z with: 2024-06-26T07:12:12.9115420Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:12.9116090Z s3-bucket: gha-artifacts 2024-06-26T07:12:12.9116618Z region: us-east-1 2024-06-26T07:12:12.9117068Z env: 2024-06-26T07:12:12.9117472Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:12.9118160Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:12.9118871Z ##[endgroup] 2024-06-26T07:12:13.3225922Z (node:13286) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:13.3226745Z 2024-06-26T07:12:13.3227032Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:13.3227766Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:13.3228717Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:13.3898246Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:12:13.3899328Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:20.8928049Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:12:20.8934044Z Artifact download has finished successfully 2024-06-26T07:12:20.9091866Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:12:20.9092305Z unzip -o artifacts.zip 2024-06-26T07:12:20.9099924Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:20.9100430Z env: 2024-06-26T07:12:20.9100707Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:20.9101158Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:20.9101639Z ##[endgroup] 2024-06-26T07:12:20.9128054Z Archive: artifacts.zip 2024-06-26T07:12:20.9129776Z creating: dist/ 2024-06-26T07:12:22.8846921Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:12:22.8847605Z creating: build/custom_test_artifacts/ 2024-06-26T07:12:22.8848191Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:12:22.8848917Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:12:22.8849747Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:22.8854432Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:22.8855363Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:22.8856508Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:22.8857484Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:22.8858445Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:22.8859587Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:22.8860786Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:22.8861905Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:22.8863019Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:22.8864324Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:22.8865478Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:22.8866964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:22.8868093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:22.8869438Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:22.8870595Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:22.8871608Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:22.8872595Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:22.8909533Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:22.8947472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:22.8948895Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:22.8991509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:22.8992903Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:22.8994304Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:22.8995754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:22.8997145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:22.8998512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:22.8999879Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:22.9001247Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:22.9002674Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:22.9003939Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:22.9005153Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:22.9006358Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:22.9008008Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:22.9009242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:22.9010433Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:22.9062240Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:22.9119148Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:22.9121255Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:22.9123326Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:22.9125039Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:22.9126801Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:22.9128641Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:12:22.9130054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:12:22.9131193Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:12:22.9132287Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:12:22.9133302Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:12:22.9134364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:12:22.9135431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:12:22.9136502Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:12:22.9137558Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:12:22.9138610Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:12:22.9147746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:12:22.9275550Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:12:22.9276552Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:12:22.9277631Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:12:22.9278834Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:12:22.9279996Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:12:22.9281072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:12:22.9282242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:12:22.9283363Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:12:22.9284489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:12:22.9285614Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:12:22.9286723Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:12:22.9301794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:12:22.9381339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:12:22.9383090Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:22.9384954Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:22.9386524Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:12:22.9387825Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:12:22.9389148Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:22.9390478Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:12:22.9391716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:12:22.9392831Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:12:22.9393958Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:12:22.9499022Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:12:22.9559988Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:12:22.9561041Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:12:22.9562077Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:12:22.9563204Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:22.9568538Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:22.9569858Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:22.9571147Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:22.9572563Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:22.9573915Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:22.9575482Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:22.9577050Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:22.9578484Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:22.9579931Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:22.9581565Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:22.9583199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:22.9585014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:22.9586614Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:22.9588247Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:22.9589853Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:22.9591310Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:22.9592689Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:22.9625222Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:22.9663776Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:22.9666010Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:22.9708634Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:22.9710712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:22.9712905Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:22.9715028Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:22.9717044Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:22.9719048Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:22.9721146Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:22.9723248Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:22.9725163Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:22.9726965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:22.9728708Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:22.9730392Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:22.9732146Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:22.9733851Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:22.9735556Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:22.9781393Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:22.9839644Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:22.9841379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:22.9842911Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:22.9844161Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:22.9845465Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:22.9846831Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:12:22.9848379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:12:22.9850088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:12:22.9851721Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:12:22.9853211Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:12:22.9854788Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:12:22.9856334Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:12:22.9858100Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:12:22.9859690Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:12:22.9861243Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:12:22.9869963Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:12:22.9932108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:12:22.9933831Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:22.9935376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:22.9936753Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:12:22.9938020Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:12:22.9939286Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:22.9940633Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:12:22.9941846Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:12:22.9942932Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:12:22.9944548Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:12:22.9989345Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:12:22.9990206Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:12:22.9990962Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:12:22.9991838Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:12:22.9996772Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:12:22.9997770Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:12:22.9998757Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:12:22.9999799Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:12:23.0000893Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:12:23.0002155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:12:23.0003312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:12:23.0004354Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:12:23.0005388Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:12:23.0006572Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:12:23.0007755Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:12:23.0008998Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:12:23.0010192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:12:23.0011390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:12:23.0012584Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:12:23.0013807Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:12:23.0014852Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:12:23.0048037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:12:23.0086910Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:12:23.0088510Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:12:23.0130811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:12:23.0132254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:12:23.0133731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:12:23.0135233Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:12:23.0136687Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:12:23.0138125Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:12:23.0139551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:12:23.0140974Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:12:23.0142360Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:12:23.0144016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:12:23.0145307Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:12:23.0146556Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:12:23.0147818Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:12:23.0149042Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:12:23.0150356Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:12:23.0197271Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:12:23.0255095Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:12:23.0256381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:12:23.0257417Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:12:23.0258327Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:12:23.0276469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:12:23.0277685Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:12:23.0278824Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:12:23.0280133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:12:23.0281504Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:12:23.0282758Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:12:23.0283927Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:12:23.0285176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:12:23.0286362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:12:23.0287533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:12:23.0288691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:12:23.0289924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:12:23.0384540Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:12:23.0385697Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:12:23.0386863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:12:23.0388156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:12:23.0389402Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:12:23.0390576Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:12:23.0391793Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:12:23.0393009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:12:23.0394226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:12:23.0395436Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:12:23.0396632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:12:23.0408430Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:12:23.0461588Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:12:23.0462880Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:12:23.0464966Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:12:23.0466035Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:12:23.0467010Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:12:23.0468019Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:12:23.0468983Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:12:23.0469877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:12:23.0470709Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:12:23.0471558Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:12:23.0567424Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:12:23.0608560Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:12:23.0609185Z creating: build/lib/ 2024-06-26T07:12:23.0693800Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:12:23.1133180Z inflating: build/lib/libprotobuf.a 2024-06-26T07:12:23.1142087Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:12:23.1149991Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:12:23.1157567Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:12:23.1158065Z inflating: build/lib/libclog.a 2024-06-26T07:12:23.1175495Z inflating: build/lib/libnnpack.a 2024-06-26T07:12:23.1177498Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:12:23.1239583Z inflating: build/lib/libgtest.a 2024-06-26T07:12:23.1312002Z inflating: build/lib/libbenchmark.a 2024-06-26T07:12:23.1373743Z inflating: build/lib/libasmjit.a 2024-06-26T07:12:23.1381329Z inflating: build/lib/libittnotify.a 2024-06-26T07:12:23.1408654Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:12:23.1530942Z inflating: build/lib/libgloo.a 2024-06-26T07:12:23.1552038Z inflating: build/lib/libfmt.a 2024-06-26T07:12:23.1640435Z inflating: build/lib/libc10.so 2024-06-26T07:12:23.1642000Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:12:23.1642484Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:12:23.1643529Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:12:23.1662264Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:12:23.1679754Z inflating: build/lib/libgmock.a 2024-06-26T07:12:23.1680200Z inflating: build/lib/libgtest_main.a 2024-06-26T07:12:23.1680680Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:12:23.2170527Z inflating: build/lib/libprotoc.a 2024-06-26T07:12:23.2542782Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:12:24.2279234Z inflating: build/lib/libdnnl.a 2024-06-26T07:12:24.2830453Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:12:24.2883482Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:12:24.2884693Z inflating: build/lib/libgmock_main.a 2024-06-26T07:12:24.4085082Z inflating: build/lib/libfbgemm.a 2024-06-26T07:12:24.4330238Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:12:24.4830987Z inflating: build/lib/libkineto.a 2024-06-26T07:12:24.5009283Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:12:24.5050219Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:12:24.5714686Z inflating: build/lib/libonnx.a 2024-06-26T07:12:26.8991844Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:12:26.8995764Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:12:26.8999624Z inflating: build/lib/libshm.so 2024-06-26T07:12:28.8874718Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:12:28.8875389Z inflating: build/lib/libtorch.so 2024-06-26T07:12:29.6805487Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:12:29.6807525Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:12:29.8701917Z inflating: build/lib/libtorch_python.so 2024-06-26T07:12:29.8769201Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:12:29.8788428Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:12:29.8812850Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:12:29.8846120Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:12:29.8846672Z creating: build/bin/ 2024-06-26T07:12:29.8894925Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:12:29.8944711Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:12:29.8993413Z inflating: build/bin/c10_Device_test 2024-06-26T07:12:29.9050777Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:12:29.9102494Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:12:29.9150840Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:12:29.9203893Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:12:29.9253197Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:12:29.9308067Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:12:29.9362911Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:12:29.9413902Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:12:29.9483166Z inflating: build/bin/c10_cow_test 2024-06-26T07:12:29.9530447Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:12:29.9578729Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:12:29.9627922Z inflating: build/bin/c10_Half_test 2024-06-26T07:12:29.9682346Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:12:29.9735674Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:12:29.9785254Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:12:29.9838229Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:12:29.9888509Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:12:29.9937764Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:12:29.9985472Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:12:30.0035586Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:12:30.0089481Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:12:30.0138399Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:12:30.0193011Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:12:30.0247761Z inflating: build/bin/c10_complex_test 2024-06-26T07:12:30.0297971Z inflating: build/bin/c10_exception_test 2024-06-26T07:12:30.0347505Z inflating: build/bin/c10_flags_test 2024-06-26T07:12:30.0395214Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:12:30.0554386Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:12:30.0605729Z inflating: build/bin/c10_irange_test 2024-06-26T07:12:30.0656914Z inflating: build/bin/c10_lazy_test 2024-06-26T07:12:30.0712181Z inflating: build/bin/c10_logging_test 2024-06-26T07:12:30.0784821Z inflating: build/bin/c10_optional_test 2024-06-26T07:12:30.0845847Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:12:30.0897817Z inflating: build/bin/c10_registry_test 2024-06-26T07:12:30.1043562Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:12:30.1093378Z inflating: build/bin/c10_ssize_test 2024-06-26T07:12:30.1143896Z inflating: build/bin/c10_string_util_test 2024-06-26T07:12:30.1200581Z inflating: build/bin/c10_string_view_test 2024-06-26T07:12:30.1249598Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:12:30.1296490Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:12:30.1350768Z inflating: build/bin/c10_typeid_test 2024-06-26T07:12:30.1783225Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:12:30.2215368Z inflating: build/bin/protoc 2024-06-26T07:12:30.2267716Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:12:30.2318281Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:12:30.2370336Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:12:30.2420796Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:12:30.2471971Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:12:30.2523195Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:12:30.2570857Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:12:30.2621950Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:12:30.2940342Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:12:30.3272094Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:12:30.3617450Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:12:30.3669129Z inflating: build/bin/FileStoreTest 2024-06-26T07:12:30.3722787Z inflating: build/bin/TCPStoreTest 2024-06-26T07:12:30.3774429Z inflating: build/bin/HashStoreTest 2024-06-26T07:12:30.3787524Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:12:30.3839700Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:12:30.3843786Z inflating: build/bin/torch_shm_manager 2024-06-26T07:12:30.3846408Z inflating: build/bin/example_allreduce 2024-06-26T07:12:30.3899467Z inflating: build/bin/test_dist_autograd 2024-06-26T07:12:30.3966862Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:12:30.3969830Z inflating: build/bin/parallel_benchmark 2024-06-26T07:12:30.4034294Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:12:30.4042826Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:12:30.4375435Z inflating: build/bin/test_lazy 2024-06-26T07:12:30.5522023Z inflating: build/bin/test_api 2024-06-26T07:12:30.5593484Z inflating: build/bin/Dict_test 2024-06-26T07:12:30.5644719Z inflating: build/bin/Dimname_test 2024-06-26T07:12:30.5706694Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:12:30.5762699Z inflating: build/bin/NamedTensor_test 2024-06-26T07:12:30.5819433Z inflating: build/bin/apply_utils_test 2024-06-26T07:12:30.5876629Z inflating: build/bin/atest 2024-06-26T07:12:30.5937938Z inflating: build/bin/basic 2024-06-26T07:12:30.5992209Z inflating: build/bin/broadcast_test 2024-06-26T07:12:30.6041320Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:12:30.6097773Z inflating: build/bin/cpu_generator_test 2024-06-26T07:12:30.6149787Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:12:30.6239647Z inflating: build/bin/cpu_rng_test 2024-06-26T07:12:30.6288149Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:12:30.6336963Z inflating: build/bin/dlconvertor_test 2024-06-26T07:12:30.6393399Z inflating: build/bin/extension_backend_test 2024-06-26T07:12:30.6447295Z inflating: build/bin/half_test 2024-06-26T07:12:30.6539617Z inflating: build/bin/ivalue_test 2024-06-26T07:12:30.6588273Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:12:30.6640392Z inflating: build/bin/math_kernel_test 2024-06-26T07:12:30.6693142Z inflating: build/bin/memory_format_test 2024-06-26T07:12:30.6745262Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:12:30.6795869Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:12:30.6850273Z inflating: build/bin/native_test 2024-06-26T07:12:30.6899299Z inflating: build/bin/operator_name_test 2024-06-26T07:12:30.6949277Z inflating: build/bin/operators_test 2024-06-26T07:12:30.6999491Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:12:30.7064502Z inflating: build/bin/pow_test 2024-06-26T07:12:30.7119761Z inflating: build/bin/quantized_test 2024-06-26T07:12:30.7169013Z inflating: build/bin/reduce_ops_test 2024-06-26T07:12:30.7217793Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:12:30.7272515Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:12:30.7330349Z inflating: build/bin/scalar_test 2024-06-26T07:12:30.7380440Z inflating: build/bin/StorageUtils_test 2024-06-26T07:12:30.7431467Z inflating: build/bin/stride_properties_test 2024-06-26T07:12:30.7509306Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:12:30.7561494Z inflating: build/bin/test_parallel 2024-06-26T07:12:30.7565089Z inflating: build/bin/thread_init_test 2024-06-26T07:12:30.7618634Z inflating: build/bin/type_ptr_test 2024-06-26T07:12:30.7676656Z inflating: build/bin/type_test 2024-06-26T07:12:30.7727206Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:12:30.7728245Z inflating: build/bin/verify_api_visibility 2024-06-26T07:12:30.7795602Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:12:30.7845715Z inflating: build/bin/weakref_test 2024-06-26T07:12:30.7895454Z inflating: build/bin/wrapdim_test 2024-06-26T07:12:30.7945587Z inflating: build/bin/xla_tensor_test 2024-06-26T07:12:30.8004276Z inflating: build/bin/IListRef_test 2024-06-26T07:12:30.8106513Z inflating: build/bin/List_test 2024-06-26T07:12:30.8171013Z inflating: build/bin/KernelFunction_test 2024-06-26T07:12:30.8289020Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:12:30.8382310Z inflating: build/bin/kernel_function_test 2024-06-26T07:12:30.8506754Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:12:30.8608135Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:12:30.8666420Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:12:30.8759376Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:12:30.8808692Z inflating: build/bin/CppSignature_test 2024-06-26T07:12:30.8862653Z inflating: build/bin/backend_fallback_test 2024-06-26T07:12:30.8910086Z inflating: build/bin/op_allowlist_test 2024-06-26T07:12:30.9201729Z inflating: build/bin/op_registration_test 2024-06-26T07:12:30.9264334Z inflating: build/bin/inline_container_test 2024-06-26T07:12:30.9313983Z inflating: build/bin/cuda_apply_test 2024-06-26T07:12:30.9363421Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:12:30.9420555Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:12:30.9473532Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:12:30.9541626Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:12:30.9599178Z inflating: build/bin/cuda_complex_test 2024-06-26T07:12:30.9647271Z inflating: build/bin/cuda_device_test 2024-06-26T07:12:30.9703217Z inflating: build/bin/cuda_cub_test 2024-06-26T07:12:30.9752041Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:12:30.9814881Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:12:30.9869904Z inflating: build/bin/cuda_generator_test 2024-06-26T07:12:30.9918172Z inflating: build/bin/cuda_half_test 2024-06-26T07:12:30.9969301Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:12:31.0016105Z inflating: build/bin/cuda_optional_test 2024-06-26T07:12:31.0065959Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:12:31.0116799Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:12:31.0166074Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:12:31.0225040Z inflating: build/bin/cuda_stream_test 2024-06-26T07:12:31.0272801Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:12:31.0323928Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:12:31.0337534Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:12:31.0401047Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:12:31.0456605Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:12:31.0517980Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:12:31.0578099Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:12:31.1381967Z inflating: build/bin/test_tensorexpr 2024-06-26T07:12:31.1941469Z inflating: build/bin/test_jit 2024-06-26T07:12:31.1941974Z creating: .additional_ci_files/ 2024-06-26T07:12:31.1995197Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:12:31.2205027Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:12:31.2230418Z ##[group]Run rm artifacts.zip 2024-06-26T07:12:31.2230802Z rm artifacts.zip 2024-06-26T07:12:31.2240027Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:31.2240528Z env: 2024-06-26T07:12:31.2240821Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.2241274Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.2241760Z ##[endgroup] 2024-06-26T07:12:31.2952148Z ##[group]Run df -H 2024-06-26T07:12:31.2952468Z df -H 2024-06-26T07:12:31.2959704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:31.2960197Z env: 2024-06-26T07:12:31.2960474Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.2961055Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.2961538Z ##[endgroup] 2024-06-26T07:12:31.2987207Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:12:31.2987748Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:12:31.2988234Z tmpfs 34G 82k 34G 1% /dev/shm 2024-06-26T07:12:31.2988690Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:12:31.2989164Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:12:31.2989636Z /dev/nvme0n1p1 162G 44G 118G 27% / 2024-06-26T07:12:31.3027592Z Prepare all required actions 2024-06-26T07:12:31.3028043Z Getting action download info 2024-06-26T07:12:31.4185490Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:12:31.4185949Z with: 2024-06-26T07:12:31.4186202Z env: 2024-06-26T07:12:31.4186476Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.4186914Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.4187387Z ##[endgroup] 2024-06-26T07:12:31.4218193Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:12:31.4218628Z with: 2024-06-26T07:12:31.4218884Z name: td_results 2024-06-26T07:12:31.4219197Z s3-bucket: gha-artifacts 2024-06-26T07:12:31.4219541Z region: us-east-1 2024-06-26T07:12:31.4219829Z env: 2024-06-26T07:12:31.4220094Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.4220538Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.4221009Z ##[endgroup] 2024-06-26T07:12:31.8369495Z (node:13329) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:12:31.8370233Z 2024-06-26T07:12:31.8370501Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:12:31.8371218Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:12:31.8372054Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:12:31.9074630Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:12:31.9081232Z Artifact download has finished successfully 2024-06-26T07:12:31.9209793Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:12:31.9210283Z mkdir -p .additional_ci_files 2024-06-26T07:12:31.9210845Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:12:31.9218563Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:31.9219053Z env: 2024-06-26T07:12:31.9219343Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.9219814Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.9220298Z ##[endgroup] 2024-06-26T07:12:31.9254463Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:12:31.9264530Z ##[error]Process completed with exit code 1. 2024-06-26T07:12:31.9310626Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:12:31.9311131Z .github/scripts/parse_ref.py 2024-06-26T07:12:31.9318219Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:31.9318600Z env: 2024-06-26T07:12:31.9318886Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.9319333Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.9319822Z ##[endgroup] 2024-06-26T07:12:31.9545107Z Prepare all required actions 2024-06-26T07:12:31.9581394Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:12:31.9581853Z with: 2024-06-26T07:12:31.9582391Z github-token: *** 2024-06-26T07:12:31.9582694Z env: 2024-06-26T07:12:31.9582962Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.9583651Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.9584141Z ##[endgroup] 2024-06-26T07:12:31.9598771Z ##[group]Run set -eux 2024-06-26T07:12:31.9599103Z set -eux 2024-06-26T07:12:31.9599683Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:12:31.9607347Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:31.9607833Z env: 2024-06-26T07:12:31.9608114Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:31.9608708Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:31.9609340Z GITHUB_TOKEN: *** 2024-06-26T07:12:31.9609642Z ##[endgroup] 2024-06-26T07:12:31.9633420Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-08e7836a94a47ea58 2024-06-26T07:12:32.5475023Z setting job-id=26688916330 2024-06-26T07:12:32.5476032Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:32.5668434Z Prepare all required actions 2024-06-26T07:12:32.5668876Z Getting action download info 2024-06-26T07:12:32.6876329Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:12:32.6876800Z with: 2024-06-26T07:12:32.6877266Z github-token: *** 2024-06-26T07:12:32.6883858Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:12:32.6891174Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:32.6892083Z env: 2024-06-26T07:12:32.6892390Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:32.6892838Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:32.6893317Z ##[endgroup] 2024-06-26T07:12:32.6933159Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:12:32.6933703Z with: 2024-06-26T07:12:32.6933968Z shell: bash 2024-06-26T07:12:32.6934262Z timeout_minutes: 10 2024-06-26T07:12:32.6934589Z max_attempts: 5 2024-06-26T07:12:32.6934892Z retry_wait_seconds: 30 2024-06-26T07:12:32.6935982Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:32.6937127Z polling_interval_seconds: 1 2024-06-26T07:12:32.6937491Z warning_on_retry: true 2024-06-26T07:12:32.6937825Z continue_on_error: false 2024-06-26T07:12:32.6938158Z env: 2024-06-26T07:12:32.6938425Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:32.6938866Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:32.6939485Z GITHUB_TOKEN: *** 2024-06-26T07:12:32.6939805Z ##[endgroup] 2024-06-26T07:12:32.7434395Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:12:32.9573366Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:12:32.9721741Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:12:32.9865559Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:12:32.9874982Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:12:32.9894932Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:12:32.9903006Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:12:33.0100812Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:12:33.7428054Z Command completed after 1 attempt(s). 2024-06-26T07:12:33.7469085Z ##[group]Run set -x 2024-06-26T07:12:33.7469441Z set -x 2024-06-26T07:12:33.7469743Z  2024-06-26T07:12:33.7470291Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:33.7470963Z # in runner workspace 2024-06-26T07:12:33.7471496Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:12:33.7479352Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:33.7479852Z env: 2024-06-26T07:12:33.7480139Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:33.7480597Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:33.7481082Z ##[endgroup] 2024-06-26T07:12:33.7504676Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:12:33.7701031Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:33.7701573Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:12:33.7702054Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:12:33.7702452Z  2024-06-26T07:12:33.7702985Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:12:33.7703844Z # in runner workspace 2024-06-26T07:12:33.7704410Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:12:33.7705045Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:12:33.7705501Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:12:33.7712131Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:12:33.7720482Z  --selected-test-configs "" \ 2024-06-26T07:12:33.7720962Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:12:33.7721401Z  --tag "${TAG}" \ 2024-06-26T07:12:33.7721808Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:12:33.7722305Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:12:33.7722838Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:12:33.7730237Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:33.7730734Z env: 2024-06-26T07:12:33.7731005Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:33.7731454Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:33.7732116Z GITHUB_TOKEN: *** 2024-06-26T07:12:33.7732924Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:33.7733800Z PR_NUMBER: 2024-06-26T07:12:33.7734109Z TAG: ciflow/inductor/129470 2024-06-26T07:12:33.7734473Z EVENT_NAME: push 2024-06-26T07:12:33.7734775Z SCHEDULE: 2024-06-26T07:12:33.7735053Z HEAD_BRANCH: 2024-06-26T07:12:33.7735343Z ##[endgroup] 2024-06-26T07:12:33.7756155Z Workflow: inductor-periodic 2024-06-26T07:12:33.7757419Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:33.9792601Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:12:34.1181022Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:12:34.1181483Z echo "Filtered matrix:" 2024-06-26T07:12:34.1188173Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:12:34.1194947Z  2024-06-26T07:12:34.1195221Z echo 2024-06-26T07:12:34.1195598Z echo "Is the current job unstable? False" 2024-06-26T07:12:34.1196055Z  2024-06-26T07:12:34.1196328Z echo 2024-06-26T07:12:34.1196683Z echo "Is keep-going label set? False" 2024-06-26T07:12:34.1197123Z  2024-06-26T07:12:34.1197396Z echo 2024-06-26T07:12:34.1197719Z echo "Renabled issues? " 2024-06-26T07:12:34.1205501Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:34.1205994Z env: 2024-06-26T07:12:34.1206276Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:34.1206845Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:34.1207324Z ##[endgroup] 2024-06-26T07:12:34.1226483Z Filtered matrix: 2024-06-26T07:12:34.1235099Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:12:34.1241695Z 2024-06-26T07:12:34.1241844Z Is the current job unstable? False 2024-06-26T07:12:34.1242201Z 2024-06-26T07:12:34.1242402Z Is keep-going label set? False 2024-06-26T07:12:34.1242701Z 2024-06-26T07:12:34.1242834Z Renabled issues? 2024-06-26T07:12:34.1283929Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:34.1284650Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:12:34.1292352Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:34.1292863Z env: 2024-06-26T07:12:34.1293146Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:34.1293599Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:34.1294088Z JOB_TIMEOUT: 240 2024-06-26T07:12:34.1294394Z ##[endgroup] 2024-06-26T07:12:34.1361757Z ##[group]Run set -x 2024-06-26T07:12:34.1362226Z set -x 2024-06-26T07:12:34.1362532Z  2024-06-26T07:12:34.1362899Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:12:34.1363435Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:12:34.1363999Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:12:34.1364508Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:12:34.1364924Z else 2024-06-26T07:12:34.1365267Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:34.1365705Z fi 2024-06-26T07:12:34.1365990Z  2024-06-26T07:12:34.1366459Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:12:34.1367206Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:12:34.1367881Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:12:34.1368465Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:12:34.1368927Z container_name=$(docker run \ 2024-06-26T07:12:34.1369351Z  ${GPU_FLAG:-} \ 2024-06-26T07:12:34.1369730Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:12:34.1370134Z  -e PR_NUMBER \ 2024-06-26T07:12:34.1370495Z  -e GITHUB_ACTIONS \ 2024-06-26T07:12:34.1370894Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:12:34.1371303Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:12:34.1371693Z  -e GITHUB_JOB \ 2024-06-26T07:12:34.1372054Z  -e GITHUB_RUN_ID \ 2024-06-26T07:12:34.1372559Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:12:34.1372971Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:12:34.1373364Z  -e JOB_ID \ 2024-06-26T07:12:34.1373707Z  -e JOB_NAME \ 2024-06-26T07:12:34.1374056Z  -e BASE_SHA \ 2024-06-26T07:12:34.1374400Z  -e BRANCH \ 2024-06-26T07:12:34.1374730Z  -e SHA1 \ 2024-06-26T07:12:34.1375086Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:12:34.1375497Z  -e IN_WHEEL_TEST \ 2024-06-26T07:12:34.1375884Z  -e SHARD_NUMBER \ 2024-06-26T07:12:34.1376255Z  -e TEST_CONFIG \ 2024-06-26T07:12:34.1376630Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:12:34.1377023Z  -e REENABLED_ISSUES \ 2024-06-26T07:12:34.1377441Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:12:34.1377863Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:12:34.1378266Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:12:34.1378650Z  -e NO_TD \ 2024-06-26T07:12:34.1378994Z  -e TD_DISTRIBUTED \ 2024-06-26T07:12:34.1379380Z  -e PR_LABELS \ 2024-06-26T07:12:34.1379790Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:12:34.1380247Z  -e SCCACHE_BUCKET \ 2024-06-26T07:12:34.1380639Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:12:34.1381055Z  -e XLA_CUDA \ 2024-06-26T07:12:34.1381466Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:12:34.1381983Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:12:34.1382555Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:12:34.1383072Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:12:34.1383675Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:12:34.1384101Z  -e DASHBOARD_TAG \ 2024-06-26T07:12:34.1384576Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:12:34.1385135Z  --security-opt seccomp=unconfined \ 2024-06-26T07:12:34.1385600Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:12:34.1386005Z  --ipc=host \ 2024-06-26T07:12:34.1386362Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:12:34.1386759Z  --tty \ 2024-06-26T07:12:34.1387078Z  --detach \ 2024-06-26T07:12:34.1387441Z  --name="${container_name}" \ 2024-06-26T07:12:34.1387862Z  --user jenkins \ 2024-06-26T07:12:34.1388438Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:12:34.1389008Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:12:34.1389448Z  "${DOCKER_IMAGE}" 2024-06-26T07:12:34.1389801Z ) 2024-06-26T07:12:34.1390230Z # Propagate download.pytorch.org IP to container 2024-06-26T07:12:34.1391166Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:12:34.1392136Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:12:34.1393044Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:12:34.1400264Z shell: /usr/bin/bash -e {0} 2024-06-26T07:12:34.1400620Z env: 2024-06-26T07:12:34.1400911Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:34.1401365Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:34.1402086Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:34.1402598Z PR_NUMBER: 2024-06-26T07:12:34.1402930Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:12:34.1403362Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:12:34.1403766Z GITHUB_JOB: test 2024-06-26T07:12:34.1404087Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:12:34.1404454Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:12:34.1404809Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:12:34.1405146Z JOB_ID: 26688916330 2024-06-26T07:12:34.1405972Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:34.1406926Z BRANCH: 2024-06-26T07:12:34.1407261Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:34.1407777Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:34.1408257Z TEST_CONFIG: aot_eager_torchbench 2024-06-26T07:12:34.1408658Z SHARD_NUMBER: 2 2024-06-26T07:12:34.1408969Z NUM_TEST_SHARDS: 2 2024-06-26T07:12:34.1409300Z REENABLED_ISSUES: 2024-06-26T07:12:34.1409650Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:12:34.1410044Z VERBOSE_TEST_LOGS: False 2024-06-26T07:12:34.1410410Z NO_TEST_TIMEOUT: False 2024-06-26T07:12:34.1410754Z NO_TD: False 2024-06-26T07:12:34.1411066Z TD_DISTRIBUTED: False 2024-06-26T07:12:34.1411483Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:12:34.1411999Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:12:34.1412474Z SHM_SIZE: 2g 2024-06-26T07:12:34.1413541Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:34.1414698Z XLA_CUDA: 2024-06-26T07:12:34.1415189Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:34.1415799Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:12:34.1416237Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:12:34.1416643Z DASHBOARD_TAG: 2024-06-26T07:12:34.1417156Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:12:34.1417527Z ##[endgroup] 2024-06-26T07:12:34.1436425Z + [[ aot_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:12:34.1437330Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:12:34.1437863Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:12:34.1444322Z +++ nproc --ignore=2 2024-06-26T07:12:34.1470230Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:45.3626374Z + container_name=255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T07:12:45.3627179Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:12:45.3629275Z + docker exec -i 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:12:45.4222389Z + echo DOCKER_CONTAINER_ID=255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T07:12:45.4224640Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:12:45.4226391Z + docker exec -t 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:12:45.7992393Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:12:46.1131236Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:12:46.1136971Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:12:46.1139127Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:12:46.1141254Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:12:46.1143720Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:12:46.1146122Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:12:46.1172363Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:12:46.1230355Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:12:46.1664837Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:12:46.1825796Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:12:47.4925566Z Installing collected packages: torch 2024-06-26T07:12:56.2578450Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:12:56.2579822Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:12:56.2580491Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:12:56.3411308Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:12:56.3415931Z + source .ci/pytorch/common.sh 2024-06-26T07:12:56.3418013Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:12:56.3423773Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:12:56.3425122Z +++ declare -f -t trap_add 2024-06-26T07:12:56.3431153Z ++ set -ex 2024-06-26T07:12:56.3431813Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:12:56.3432482Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:12:56.3433123Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:12:56.3435220Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:12:56.3443696Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:12:56.3444244Z + trap_add cleanup_workspace EXIT 2024-06-26T07:12:56.3444666Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:12:56.3445042Z + shift 2024-06-26T07:12:56.3445333Z + for trap_add_name in "$@" 2024-06-26T07:12:56.3450156Z +++ trap -p EXIT 2024-06-26T07:12:56.3451028Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:12:56.3451531Z +++ extract_trap_cmd 2024-06-26T07:12:56.3452004Z +++ printf '%s\n' '' 2024-06-26T07:12:56.3452390Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:12:56.3452949Z + trap -- ' 2024-06-26T07:12:56.3453288Z cleanup_workspace' EXIT 2024-06-26T07:12:56.3453748Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:12:56.8092137Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:12:56.8105119Z + echo 'Environment variables:' 2024-06-26T07:12:56.8105542Z Environment variables: 2024-06-26T07:12:56.8105875Z + env 2024-06-26T07:12:56.8110138Z INSTALLED_DB=yes 2024-06-26T07:12:56.8110641Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:12:56.8111164Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:12:56.8111671Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:12:56.8112504Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:12:56.8113086Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:12:56.8113567Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:12:56.8114082Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:56.8115964Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:56.8116708Z HOSTNAME=255a27a1a4ef 2024-06-26T07:12:56.8117804Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8118897Z GITHUB_ACTION=__self 2024-06-26T07:12:56.8119277Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:12:56.8124056Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:12:56.8127938Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:12:56.8128458Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:12:56.8128814Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:12:56.8129174Z TEST_CONFIG=aot_eager_torchbench 2024-06-26T07:12:56.8129605Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:12:56.8130093Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:12:56.8130596Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:12:56.8131038Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:12:56.8131463Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:12:56.8131902Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:56.8132400Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:12:56.8132864Z GITHUB_REF_TYPE=tag 2024-06-26T07:12:56.8133187Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:12:56.8133565Z NCCL_VERSION=2.17.1-1 2024-06-26T07:12:56.8134129Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8134577Z XLA_CUDA= 2024-06-26T07:12:56.8135058Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:12:56.8136819Z *** 2024-06-26T07:12:56.8137116Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:12:56.8137531Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:12:56.8137886Z GITHUB_ACTIONS=true 2024-06-26T07:12:56.8138221Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:56.8138724Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:12:56.8139255Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:12:56.8139712Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8140219Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:12:56.8140696Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8141600Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:12:56.8142358Z UCC_HOME=/usr 2024-06-26T07:12:56.8142694Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:12:56.8143099Z VERBOSE_TEST_LOGS=False 2024-06-26T07:12:56.8143638Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:12:56.8144087Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:12:56.8144585Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:12:56.8145046Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:12:56.8145413Z SHARD_NUMBER=2 2024-06-26T07:12:56.8145725Z GITHUB_REF_PROTECTED=false 2024-06-26T07:12:56.8146076Z HOME=/var/lib/jenkins 2024-06-26T07:12:56.8146442Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:12:56.8146876Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:12:56.8147353Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:12:56.8148033Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:12:56.8148448Z CUDA_VERSION=12.1.1 2024-06-26T07:12:56.8148857Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:12:56.8149343Z NUM_TEST_SHARDS=2 2024-06-26T07:12:56.8149649Z UCX_HOME=/usr 2024-06-26T07:12:56.8150154Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:12:56.8151209Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8152609Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:56.8153964Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8155062Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:12:56.8155708Z GITHUB_EVENT_NAME=push 2024-06-26T07:12:56.8156036Z DASHBOARD_TAG= 2024-06-26T07:12:56.8156347Z GITHUB_RUN_ID=9673646806 2024-06-26T07:12:56.8156795Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:12:56.8157314Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:12:56.8158290Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8159190Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:56.8159597Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:12:56.8159975Z PR_NUMBER= 2024-06-26T07:12:56.8160271Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:12:56.8160610Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:12:56.8161047Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:12:56.8161502Z TERM=xterm 2024-06-26T07:12:56.8161931Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:12:56.8162350Z INSTALLED_VISION=yes 2024-06-26T07:12:56.8162654Z BRANCH= 2024-06-26T07:12:56.8162938Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:12:56.8163346Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:12:56.8163759Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:12:56.8164499Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:12:56.8165246Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:12:56.8165723Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:12:56.8166260Z REENABLED_ISSUES= 2024-06-26T07:12:56.8166569Z SHLVL=1 2024-06-26T07:12:56.8166831Z MAX_JOBS=14 2024-06-26T07:12:56.8167146Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:12:56.8167493Z NVARCH=x86_64 2024-06-26T07:12:56.8167797Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:12:56.8168244Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8168769Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:12:56.8169242Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:12:56.8169889Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:56.8170457Z GITHUB_JOB=test 2024-06-26T07:12:56.8170841Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:12:56.8171392Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:12:56.8171910Z NO_TEST_TIMEOUT=False 2024-06-26T07:12:56.8172241Z TD_DISTRIBUTED=False 2024-06-26T07:12:56.8172610Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:12:56.8173052Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:12:56.8173464Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:12:56.8173826Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:12:56.8174162Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:12:56.8174512Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:12:56.8175533Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:56.8176577Z GITHUB_BASE_REF= 2024-06-26T07:12:56.8176900Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:12:56.8177271Z CI=true 2024-06-26T07:12:56.8177654Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:56.8178060Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:12:56.8178434Z JOB_ID=26688916330 2024-06-26T07:12:56.8178745Z INSTALLED_PROTOBUF=yes 2024-06-26T07:12:56.8179096Z GITHUB_HEAD_REF= 2024-06-26T07:12:56.8179417Z GITHUB_ACTION_REF= 2024-06-26T07:12:56.8179838Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:12:56.8180358Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:12:56.8180759Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:12:56.8181634Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8182442Z NO_TD=False 2024-06-26T07:12:56.8182745Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:12:56.8183098Z _=/usr/bin/env 2024-06-26T07:12:56.8183809Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:12:56.8255990Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:12:56.8256994Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:12:56.8257895Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:12:56.8258757Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:12:56.8259365Z + BUILD_DIR=build 2024-06-26T07:12:56.8259698Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:12:56.8260187Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:12:56.8260524Z + SHARD_NUMBER=2 2024-06-26T07:12:56.8260827Z + NUM_TEST_SHARDS=2 2024-06-26T07:12:56.8261143Z + export VALGRIND=ON 2024-06-26T07:12:56.8261459Z + VALGRIND=ON 2024-06-26T07:12:56.8261914Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:12:56.8262410Z + [[ 0 == \1 ]] 2024-06-26T07:12:56.8262715Z + [[ False == \1 ]] 2024-06-26T07:12:56.8263174Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:12:56.8264041Z ++ realpath build/custom_test_artifacts 2024-06-26T07:12:56.8269165Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:12:56.8270030Z + [[ -n '' ]] 2024-06-26T07:12:56.8270372Z + echo 'Environment variables' 2024-06-26T07:12:56.8270749Z Environment variables 2024-06-26T07:12:56.8271064Z + env 2024-06-26T07:12:56.8274998Z INSTALLED_DB=yes 2024-06-26T07:12:56.8275732Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:12:56.8276565Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:12:56.8277155Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:12:56.8277973Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:12:56.8278611Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:12:56.8279156Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:12:56.8279799Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:56.8280370Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:12:56.8280941Z HOSTNAME=255a27a1a4ef 2024-06-26T07:12:56.8282040Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8282999Z GITHUB_ACTION=__self 2024-06-26T07:12:56.8283392Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:12:56.8287337Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:12:56.8291185Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:12:56.8291824Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:12:56.8292170Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:12:56.8292526Z TEST_CONFIG=aot_eager_torchbench 2024-06-26T07:12:56.8292930Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:12:56.8293384Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:12:56.8293843Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:12:56.8294285Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:12:56.8294704Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:12:56.8295132Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:56.8295628Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:12:56.8296092Z GITHUB_REF_TYPE=tag 2024-06-26T07:12:56.8296419Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:12:56.8296785Z NCCL_VERSION=2.17.1-1 2024-06-26T07:12:56.8297169Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8297606Z XLA_CUDA= 2024-06-26T07:12:56.8298078Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:12:56.8298468Z *** 2024-06-26T07:12:56.8298754Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:12:56.8299149Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:12:56.8299512Z GITHUB_ACTIONS=true 2024-06-26T07:12:56.8299837Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:12:56.8300320Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:12:56.8300841Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:12:56.8301308Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8301798Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:12:56.8302274Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8303183Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:12:56.8304203Z UCC_HOME=/usr 2024-06-26T07:12:56.8304540Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:12:56.8304938Z VERBOSE_TEST_LOGS=False 2024-06-26T07:12:56.8305281Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:12:56.8305729Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:12:56.8306224Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:12:56.8306686Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:12:56.8307061Z SHARD_NUMBER=2 2024-06-26T07:12:56.8307362Z GITHUB_REF_PROTECTED=false 2024-06-26T07:12:56.8307714Z HOME=/var/lib/jenkins 2024-06-26T07:12:56.8308081Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:12:56.8308604Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:12:56.8309092Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:12:56.8309681Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:12:56.8310100Z CUDA_VERSION=12.1.1 2024-06-26T07:12:56.8310508Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:12:56.8310947Z NUM_TEST_SHARDS=2 2024-06-26T07:12:56.8311248Z UCX_HOME=/usr 2024-06-26T07:12:56.8311753Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:12:56.8312802Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8314185Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:12:56.8315533Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8316630Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:12:56.8317262Z GITHUB_EVENT_NAME=push 2024-06-26T07:12:56.8317597Z DASHBOARD_TAG= 2024-06-26T07:12:56.8317903Z GITHUB_RUN_ID=9673646806 2024-06-26T07:12:56.8318352Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:12:56.8318874Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:12:56.8319847Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8320740Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:12:56.8321222Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:12:56.8321589Z PR_NUMBER= 2024-06-26T07:12:56.8321969Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:12:56.8322295Z VALGRIND=ON 2024-06-26T07:12:56.8322604Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:12:56.8323045Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:12:56.8323503Z TERM=xterm 2024-06-26T07:12:56.8323842Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:12:56.8324254Z INSTALLED_VISION=yes 2024-06-26T07:12:56.8324575Z BRANCH= 2024-06-26T07:12:56.8324863Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:12:56.8325272Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:12:56.8325693Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:12:56.8326434Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:12:56.8327188Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:12:56.8327673Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:12:56.8328129Z REENABLED_ISSUES= 2024-06-26T07:12:56.8328448Z SHLVL=1 2024-06-26T07:12:56.8328713Z MAX_JOBS=14 2024-06-26T07:12:56.8329041Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:12:56.8329424Z NVARCH=x86_64 2024-06-26T07:12:56.8329732Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:12:56.8330184Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:12:56.8330706Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:12:56.8331201Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:12:56.8331855Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:12:56.8332421Z GITHUB_JOB=test 2024-06-26T07:12:56.8332815Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:12:56.8333374Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:12:56.8333886Z NO_TEST_TIMEOUT=False 2024-06-26T07:12:56.8334218Z TD_DISTRIBUTED=False 2024-06-26T07:12:56.8334585Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:12:56.8335018Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:12:56.8335441Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:12:56.8335799Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:12:56.8336148Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:12:56.8336502Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:12:56.8337529Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:56.8338641Z GITHUB_BASE_REF= 2024-06-26T07:12:56.8338975Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:12:56.8339338Z CI=true 2024-06-26T07:12:56.8339652Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:12:56.8340057Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:12:56.8340430Z JOB_ID=26688916330 2024-06-26T07:12:56.8340746Z INSTALLED_PROTOBUF=yes 2024-06-26T07:12:56.8341076Z GITHUB_HEAD_REF= 2024-06-26T07:12:56.8341388Z GITHUB_ACTION_REF= 2024-06-26T07:12:56.8341819Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:12:56.8342335Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:12:56.8342741Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:12:56.8343868Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_bbdbc852-20bd-4c0f-b674-2d1a0a4b517c 2024-06-26T07:12:56.8344685Z NO_TD=False 2024-06-26T07:12:56.8344997Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:12:56.8345356Z _=/usr/bin/env 2024-06-26T07:12:56.8345698Z + echo 'Testing pytorch' 2024-06-26T07:12:56.8346042Z Testing pytorch 2024-06-26T07:12:56.8346355Z + export LANG=C.UTF-8 2024-06-26T07:12:56.8346691Z + LANG=C.UTF-8 2024-06-26T07:12:56.8346982Z + PR_NUMBER= 2024-06-26T07:12:56.8347324Z + [[ aot_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:12:56.8347831Z + [[ aot_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:12:56.8348327Z + [[ aot_eager_torchbench == \s\l\o\w ]] 2024-06-26T07:12:56.8348947Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:12:56.8349627Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:12:56.8350164Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:12:56.8366400Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:12:56.8366912Z + [[ aot_eager_torchbench == *crossref* ]] 2024-06-26T07:12:56.8367521Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:12:56.8368144Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:12:56.8368791Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:12:56.8369355Z + pip_install --user ninja==1.10.2 2024-06-26T07:12:56.8369940Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:12:57.1951754Z Collecting ninja==1.10.2 2024-06-26T07:12:57.2113350Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:12:57.2257878Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:12:58.5122412Z Installing collected packages: ninja 2024-06-26T07:12:58.5188943Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:12:58.5190218Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:12:58.5235162Z Successfully installed ninja-1.10.2 2024-06-26T07:12:58.6134895Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:58.6137202Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:12:58.6138709Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:12:58.6139215Z + install_tlparse 2024-06-26T07:12:58.6139583Z + pip_install --user tlparse==0.3.7 2024-06-26T07:12:58.6140125Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:12:58.9664894Z Collecting tlparse==0.3.7 2024-06-26T07:12:58.9819490Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:12:58.9921091Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:13:00.3181361Z Installing collected packages: tlparse 2024-06-26T07:13:00.3543844Z Successfully installed tlparse-0.3.7 2024-06-26T07:13:00.4400667Z ++ python -m site --user-base 2024-06-26T07:13:00.4555685Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:13:00.4557499Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:13:00.4558171Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:13:00.4558827Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:00.4559844Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:13:00.4561360Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:13:00.4562418Z + cd test 2024-06-26T07:13:00.4563128Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:13:01.9276237Z + [[ aot_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:13:01.9276935Z + [[ aot_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:13:01.9277776Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:13:01.9278433Z + pushd test 2024-06-26T07:13:01.9278826Z ~/workspace/test ~/workspace 2024-06-26T07:13:01.9279361Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:13:03.4015828Z + CUDA_VERSION=12.1 2024-06-26T07:13:03.4016410Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:13:03.4016751Z + ISCUDA124= 2024-06-26T07:13:03.4017055Z + popd 2024-06-26T07:13:03.4017657Z ~/workspace 2024-06-26T07:13:03.4020206Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:13:03.4020908Z + [[ aot_eager_torchbench == *dynamo_eager* ]] 2024-06-26T07:13:03.4021437Z + [[ aot_eager_torchbench == *aot_eager* ]] 2024-06-26T07:13:03.4022005Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-06-26T07:13:03.4022491Z + [[ aot_eager_torchbench == *dynamic* ]] 2024-06-26T07:13:03.4022974Z + [[ aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:03.4023720Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:13:03.4047942Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:03.4048693Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:13:03.4049327Z + cd test 2024-06-26T07:13:03.4049924Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:13:04.7512449Z PyTorch built with: 2024-06-26T07:13:04.7513171Z - GCC 9.4 2024-06-26T07:13:04.7513617Z - C++ Version: 201703 2024-06-26T07:13:04.7514809Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:04.7516353Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:04.7517267Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:04.7517786Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:13:04.7518348Z - NNPACK is enabled 2024-06-26T07:13:04.7518737Z - CPU capability usage: AVX2 2024-06-26T07:13:04.7519239Z - CUDA Runtime 12.1 2024-06-26T07:13:04.7519824Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:13:04.7520438Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:13:04.7520953Z - Magma 2.6.1 2024-06-26T07:13:04.7528762Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:13:04.7535648Z 2024-06-26T07:13:05.0275293Z + cd test 2024-06-26T07:13:05.0275982Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:13:06.2477592Z ATen/Parallel: 2024-06-26T07:13:06.2478121Z at::get_num_threads() : 8 2024-06-26T07:13:06.2478621Z at::get_num_interop_threads() : 16 2024-06-26T07:13:06.2479173Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:13:06.2479713Z omp_get_max_threads() : 8 2024-06-26T07:13:06.2480678Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:13:06.2481473Z mkl_get_max_threads() : 8 2024-06-26T07:13:06.2482232Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:13:06.2482852Z std::thread::hardware_concurrency() : 16 2024-06-26T07:13:06.2483281Z Environment variables: 2024-06-26T07:13:06.2483627Z OMP_NUM_THREADS : [not set] 2024-06-26T07:13:06.2483995Z MKL_NUM_THREADS : [not set] 2024-06-26T07:13:06.2484359Z ATen parallel backend: OpenMP 2024-06-26T07:13:06.2484923Z 2024-06-26T07:13:06.4898578Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:13:06.4899350Z + [[ aot_eager_torchbench == *backward* ]] 2024-06-26T07:13:06.4899971Z + [[ aot_eager_torchbench == *xla* ]] 2024-06-26T07:13:06.4900584Z + [[ aot_eager_torchbench == *executorch* ]] 2024-06-26T07:13:06.4901191Z + [[ aot_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:13:06.4901885Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:13:06.4902642Z + [[ aot_eager_torchbench == distributed ]] 2024-06-26T07:13:06.4903149Z + [[ aot_eager_torchbench == *inductor_distributed* ]] 2024-06-26T07:13:06.4903964Z + [[ aot_eager_torchbench == *inductor-halide* ]] 2024-06-26T07:13:06.4904547Z + [[ aot_eager_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:13:06.4905079Z + [[ aot_eager_torchbench == *huggingface* ]] 2024-06-26T07:13:06.4905533Z + [[ aot_eager_torchbench == *timm* ]] 2024-06-26T07:13:06.4905976Z + [[ aot_eager_torchbench == *torchbench* ]] 2024-06-26T07:13:06.4906454Z + [[ aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:13:06.4906891Z + install_torchaudio cuda 2024-06-26T07:13:06.4907231Z + local commit 2024-06-26T07:13:06.4907550Z ++ get_pinned_commit audio 2024-06-26T07:13:06.4907919Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:13:06.4913258Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:06.4913808Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:13:06.4914233Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:13:06.4915138Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:06.4916540Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:06.8016098Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:06.8020858Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-lgx1hps4 2024-06-26T07:13:06.8039189Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-lgx1hps4 2024-06-26T07:13:07.4492056Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:13:07.4510176Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:07.8268740Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:07.9523437Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:13:07.9524492Z Running command git submodule update --init --recursive -q 2024-06-26T07:13:10.1023040Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:13:10.1053746Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:13:10.1117099Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:13:10.1122890Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:13:10.1126205Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:13:10.1128961Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:13:10.1132175Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:13:10.1135128Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:13:10.1563598Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:13:10.1718488Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:13:10.1789053Z Building wheels for collected packages: torchaudio 2024-06-26T07:15:35.5564061Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T07:15:35.5593179Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308818 sha256=ae8812292db2cffae2caa444d21810bb569b904f4168fda968b9634d5195c10d 2024-06-26T07:15:35.5595269Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:15:35.5620847Z Successfully built torchaudio 2024-06-26T07:15:36.8128932Z Installing collected packages: torchaudio 2024-06-26T07:15:37.0094436Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:15:37.1406269Z + install_torchtext 2024-06-26T07:15:37.1406653Z + local data_commit 2024-06-26T07:15:37.1406976Z + local text_commit 2024-06-26T07:15:37.1409222Z ++ get_pinned_commit data 2024-06-26T07:15:37.1415952Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:15:37.1420472Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:37.1422276Z ++ get_pinned_commit text 2024-06-26T07:15:37.1422694Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:15:37.1431246Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:37.1432314Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:37.1433921Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:37.4579171Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:37.4583149Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-7d9p8dcy 2024-06-26T07:15:37.4600420Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-7d9p8dcy 2024-06-26T07:15:37.9050117Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:15:37.9068842Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:38.1347859Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:38.2660729Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:15:38.2661851Z Running command git submodule update --init --recursive -q 2024-06-26T07:16:50.6860944Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:16:50.6912680Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:16:50.6915209Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:16:50.6921101Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:16:50.6981841Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:16:50.6987323Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:16:50.6990221Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:16:50.6993055Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:16:50.6995992Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:16:50.6998272Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:16:50.7188036Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:16:50.7194128Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:16:50.7200049Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:16:50.7665378Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:16:50.7830897Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:16:50.7912070Z Building wheels for collected packages: torchdata 2024-06-26T07:16:52.5265904Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:16:52.5273719Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=7a31b95015d4716b76e0249ab0f29272b76aa12fc0618d9e9cd968eaec6cf917 2024-06-26T07:16:52.5276361Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:16:52.5300242Z Successfully built torchdata 2024-06-26T07:16:53.7143893Z Installing collected packages: torchdata 2024-06-26T07:16:53.7984731Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:16:56.3062375Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:56.3064024Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:56.6171894Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:56.6175562Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-k2f2v6a7 2024-06-26T07:16:56.6193211Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-k2f2v6a7 2024-06-26T07:16:57.3968002Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:16:57.3986693Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:57.8545466Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:58.0284556Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:16:58.0285601Z Running command git submodule update --init --recursive -q 2024-06-26T07:17:05.0863109Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:17:05.0913801Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:17:05.0916922Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:17:05.0920222Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:05.0923261Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:17:05.0926796Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:17:05.0989310Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:17:05.0993747Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:17:05.0999820Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:17:05.1005401Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:17:05.1061048Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:17:05.1068127Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:17:05.1071242Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:17:05.1074646Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:17:05.1078054Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:17:05.1081233Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:17:05.1753574Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:17:05.1916581Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:17:05.1997047Z Building wheels for collected packages: torchtext 2024-06-26T07:17:41.4884117Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:17:41.4910444Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045301 sha256=484b6850960af202072211f4e7226af003e430ec63395355e9febad83146f5d2 2024-06-26T07:17:41.4913412Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:17:41.4935100Z Successfully built torchtext 2024-06-26T07:17:42.6820796Z Installing collected packages: torchtext 2024-06-26T07:17:42.7800655Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:17:42.9024683Z + install_torchvision 2024-06-26T07:17:42.9025170Z + local orig_preload 2024-06-26T07:17:42.9025591Z + local commit 2024-06-26T07:17:42.9027006Z ++ get_pinned_commit vision 2024-06-26T07:17:42.9027559Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:17:42.9037538Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:42.9038210Z + orig_preload= 2024-06-26T07:17:42.9038707Z + '[' -n '' ']' 2024-06-26T07:17:42.9039853Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:42.9041845Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:43.2154282Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:43.2157255Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-zmvf33yc 2024-06-26T07:17:43.2174220Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-zmvf33yc 2024-06-26T07:17:44.5814305Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:17:44.5833037Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:45.8009814Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:46.0333859Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:17:48.3593982Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:17:48.3640477Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:17:48.3643530Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:17:48.3650157Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:17:48.3852874Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:17:48.3859239Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:17:48.3862181Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:17:48.3865805Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:17:48.3868676Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:17:48.3871322Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:17:48.4294289Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:17:48.4449553Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:17:48.4523990Z Building wheels for collected packages: torchvision 2024-06-26T07:19:00.5007995Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:19:00.5034173Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112834 sha256=a643efc53c721bc64931a17754aacb41b0f2fc4c9601f77c623100770121e2bd 2024-06-26T07:19:00.5036461Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:19:00.5070501Z Successfully built torchvision 2024-06-26T07:19:01.7065056Z Installing collected packages: torchvision 2024-06-26T07:19:02.0986203Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:19:02.2076785Z + '[' -n '' ']' 2024-06-26T07:19:02.2077171Z + id=1 2024-06-26T07:19:02.2077633Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:19:02.2078434Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:19:02.6368554Z Collecting opencv-python==4.8.0.74 2024-06-26T07:19:02.6538633Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:19:02.6714593Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:19:02.6783670Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:19:04.4924632Z Installing collected packages: opencv-python 2024-06-26T07:19:05.3101103Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:19:05.4046264Z + [[ aot_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:19:05.4048135Z + [[ aot_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:19:05.4049231Z + [[ aot_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:19:05.4049788Z + checkout_install_torchbench 2024-06-26T07:19:05.4050165Z + local commit 2024-06-26T07:19:05.4050498Z ++ get_pinned_commit torchbench 2024-06-26T07:19:05.4050933Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:19:05.4061330Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:05.4062000Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:19:05.4070011Z Cloning into 'torchbench'... 2024-06-26T07:19:05.5482621Z remote: Enumerating objects: 30187, done. 2024-06-26T07:19:05.5486994Z remote: Counting objects: 0% (1/546) 2024-06-26T07:19:05.5487806Z remote: Counting objects: 1% (6/546) 2024-06-26T07:19:05.5488313Z remote: Counting objects: 2% (11/546) 2024-06-26T07:19:05.5488824Z remote: Counting objects: 3% (17/546) 2024-06-26T07:19:05.5489369Z remote: Counting objects: 4% (22/546) 2024-06-26T07:19:05.5490306Z remote: Counting objects: 5% (28/546) 2024-06-26T07:19:05.5490809Z remote: Counting objects: 6% (33/546) 2024-06-26T07:19:05.5491314Z remote: Counting objects: 7% (39/546) 2024-06-26T07:19:05.5494648Z remote: Counting objects: 8% (44/546) 2024-06-26T07:19:05.5495316Z remote: Counting objects: 9% (50/546) 2024-06-26T07:19:05.5495928Z remote: Counting objects: 10% (55/546) 2024-06-26T07:19:05.5496603Z remote: Counting objects: 11% (61/546) 2024-06-26T07:19:05.5497283Z remote: Counting objects: 12% (66/546) 2024-06-26T07:19:05.5498072Z remote: Counting objects: 13% (71/546) 2024-06-26T07:19:05.5498604Z remote: Counting objects: 14% (77/546) 2024-06-26T07:19:05.5499333Z remote: Counting objects: 15% (82/546) 2024-06-26T07:19:05.5500042Z remote: Counting objects: 16% (88/546) 2024-06-26T07:19:05.5500701Z remote: Counting objects: 17% (93/546) 2024-06-26T07:19:05.5501352Z remote: Counting objects: 18% (99/546) 2024-06-26T07:19:05.5502041Z remote: Counting objects: 19% (104/546) 2024-06-26T07:19:05.5502729Z remote: Counting objects: 20% (110/546) 2024-06-26T07:19:05.5503599Z remote: Counting objects: 21% (115/546) 2024-06-26T07:19:05.5504149Z remote: Counting objects: 22% (121/546) 2024-06-26T07:19:05.5504655Z remote: Counting objects: 23% (126/546) 2024-06-26T07:19:05.5505394Z remote: Counting objects: 24% (132/546) 2024-06-26T07:19:05.5505899Z remote: Counting objects: 25% (137/546) 2024-06-26T07:19:05.5506392Z remote: Counting objects: 26% (142/546) 2024-06-26T07:19:05.5506896Z remote: Counting objects: 27% (148/546) 2024-06-26T07:19:05.5507404Z remote: Counting objects: 28% (153/546) 2024-06-26T07:19:05.5507895Z remote: Counting objects: 29% (159/546) 2024-06-26T07:19:05.5508401Z remote: Counting objects: 30% (164/546) 2024-06-26T07:19:05.5508896Z remote: Counting objects: 31% (170/546) 2024-06-26T07:19:05.5509444Z remote: Counting objects: 32% (175/546) 2024-06-26T07:19:05.5509937Z remote: Counting objects: 33% (181/546) 2024-06-26T07:19:05.5510430Z remote: Counting objects: 34% (186/546) 2024-06-26T07:19:05.5510924Z remote: Counting objects: 35% (192/546) 2024-06-26T07:19:05.5511421Z remote: Counting objects: 36% (197/546) 2024-06-26T07:19:05.5511911Z remote: Counting objects: 37% (203/546) 2024-06-26T07:19:05.5512415Z remote: Counting objects: 38% (208/546) 2024-06-26T07:19:05.5512908Z remote: Counting objects: 39% (213/546) 2024-06-26T07:19:05.5513402Z remote: Counting objects: 40% (219/546) 2024-06-26T07:19:05.5513888Z remote: Counting objects: 41% (224/546) 2024-06-26T07:19:05.5514384Z remote: Counting objects: 42% (230/546) 2024-06-26T07:19:05.5514878Z remote: Counting objects: 43% (235/546) 2024-06-26T07:19:05.5515377Z remote: Counting objects: 44% (241/546) 2024-06-26T07:19:05.5515876Z remote: Counting objects: 45% (246/546) 2024-06-26T07:19:05.5516374Z remote: Counting objects: 46% (252/546) 2024-06-26T07:19:05.5516869Z remote: Counting objects: 47% (257/546) 2024-06-26T07:19:05.5517356Z remote: Counting objects: 48% (263/546) 2024-06-26T07:19:05.5517854Z remote: Counting objects: 49% (268/546) 2024-06-26T07:19:05.5518349Z remote: Counting objects: 50% (273/546) 2024-06-26T07:19:05.5518843Z remote: Counting objects: 51% (279/546) 2024-06-26T07:19:05.5519335Z remote: Counting objects: 52% (284/546) 2024-06-26T07:19:05.5519830Z remote: Counting objects: 53% (290/546) 2024-06-26T07:19:05.5520329Z remote: Counting objects: 54% (295/546) 2024-06-26T07:19:05.5520827Z remote: Counting objects: 55% (301/546) 2024-06-26T07:19:05.5521315Z remote: Counting objects: 56% (306/546) 2024-06-26T07:19:05.5521810Z remote: Counting objects: 57% (312/546) 2024-06-26T07:19:05.5522583Z remote: Counting objects: 58% (317/546) 2024-06-26T07:19:05.5523087Z remote: Counting objects: 59% (323/546) 2024-06-26T07:19:05.5523577Z remote: Counting objects: 60% (328/546) 2024-06-26T07:19:05.5524074Z remote: Counting objects: 61% (334/546) 2024-06-26T07:19:05.5524572Z remote: Counting objects: 62% (339/546) 2024-06-26T07:19:05.5525067Z remote: Counting objects: 63% (344/546) 2024-06-26T07:19:05.5525553Z remote: Counting objects: 64% (350/546) 2024-06-26T07:19:05.5526046Z remote: Counting objects: 65% (355/546) 2024-06-26T07:19:05.5526542Z remote: Counting objects: 66% (361/546) 2024-06-26T07:19:05.5527034Z remote: Counting objects: 67% (366/546) 2024-06-26T07:19:05.5527527Z remote: Counting objects: 68% (372/546) 2024-06-26T07:19:05.5528020Z remote: Counting objects: 69% (377/546) 2024-06-26T07:19:05.5528513Z remote: Counting objects: 70% (383/546) 2024-06-26T07:19:05.5529000Z remote: Counting objects: 71% (388/546) 2024-06-26T07:19:05.5529502Z remote: Counting objects: 72% (394/546) 2024-06-26T07:19:05.5529996Z remote: Counting objects: 73% (399/546) 2024-06-26T07:19:05.5530490Z remote: Counting objects: 74% (405/546) 2024-06-26T07:19:05.5530974Z remote: Counting objects: 75% (410/546) 2024-06-26T07:19:05.5531468Z remote: Counting objects: 76% (415/546) 2024-06-26T07:19:05.5531961Z remote: Counting objects: 77% (421/546) 2024-06-26T07:19:05.5532456Z remote: Counting objects: 78% (426/546) 2024-06-26T07:19:05.5532940Z remote: Counting objects: 79% (432/546) 2024-06-26T07:19:05.5533435Z remote: Counting objects: 80% (437/546) 2024-06-26T07:19:05.5534008Z remote: Counting objects: 81% (443/546) 2024-06-26T07:19:05.5534504Z remote: Counting objects: 82% (448/546) 2024-06-26T07:19:05.5534990Z remote: Counting objects: 83% (454/546) 2024-06-26T07:19:05.5535484Z remote: Counting objects: 84% (459/546) 2024-06-26T07:19:05.5535979Z remote: Counting objects: 85% (465/546) 2024-06-26T07:19:05.5536470Z remote: Counting objects: 86% (470/546) 2024-06-26T07:19:05.5536965Z remote: Counting objects: 87% (476/546) 2024-06-26T07:19:05.5537461Z remote: Counting objects: 88% (481/546) 2024-06-26T07:19:05.5537955Z remote: Counting objects: 89% (486/546) 2024-06-26T07:19:05.5538440Z remote: Counting objects: 90% (492/546) 2024-06-26T07:19:05.5538931Z remote: Counting objects: 91% (497/546) 2024-06-26T07:19:05.5539455Z remote: Counting objects: 92% (503/546) 2024-06-26T07:19:05.5539971Z remote: Counting objects: 93% (508/546) 2024-06-26T07:19:05.5540455Z remote: Counting objects: 94% (514/546) 2024-06-26T07:19:05.5540957Z remote: Counting objects: 95% (519/546) 2024-06-26T07:19:05.5541455Z remote: Counting objects: 96% (525/546) 2024-06-26T07:19:05.5541950Z remote: Counting objects: 97% (530/546) 2024-06-26T07:19:05.5542437Z remote: Counting objects: 98% (536/546) 2024-06-26T07:19:05.5542929Z remote: Counting objects: 99% (541/546) 2024-06-26T07:19:05.5543651Z remote: Counting objects: 100% (546/546) 2024-06-26T07:19:05.5544193Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:19:05.5544740Z remote: Compressing objects: 0% (1/319) 2024-06-26T07:19:05.5545267Z remote: Compressing objects: 1% (4/319) 2024-06-26T07:19:05.5545797Z remote: Compressing objects: 2% (7/319) 2024-06-26T07:19:05.5546322Z remote: Compressing objects: 3% (10/319) 2024-06-26T07:19:05.5546840Z remote: Compressing objects: 4% (13/319) 2024-06-26T07:19:05.5547367Z remote: Compressing objects: 5% (16/319) 2024-06-26T07:19:05.5547895Z remote: Compressing objects: 6% (20/319) 2024-06-26T07:19:05.5555001Z remote: Compressing objects: 7% (23/319) 2024-06-26T07:19:05.5555580Z remote: Compressing objects: 8% (26/319) 2024-06-26T07:19:05.5556101Z remote: Compressing objects: 9% (29/319) 2024-06-26T07:19:05.5559625Z remote: Compressing objects: 10% (32/319) 2024-06-26T07:19:05.5566547Z remote: Compressing objects: 11% (36/319) 2024-06-26T07:19:05.5567089Z remote: Compressing objects: 12% (39/319) 2024-06-26T07:19:05.5568864Z remote: Compressing objects: 13% (42/319) 2024-06-26T07:19:05.5569657Z remote: Compressing objects: 14% (45/319) 2024-06-26T07:19:05.5570191Z remote: Compressing objects: 15% (48/319) 2024-06-26T07:19:05.5570748Z remote: Compressing objects: 16% (52/319) 2024-06-26T07:19:05.5571376Z remote: Compressing objects: 17% (55/319) 2024-06-26T07:19:05.5571889Z remote: Compressing objects: 18% (58/319) 2024-06-26T07:19:05.5572405Z remote: Compressing objects: 19% (61/319) 2024-06-26T07:19:05.5572923Z remote: Compressing objects: 20% (64/319) 2024-06-26T07:19:05.5573438Z remote: Compressing objects: 21% (67/319) 2024-06-26T07:19:05.5574095Z remote: Compressing objects: 22% (71/319) 2024-06-26T07:19:05.5574666Z remote: Compressing objects: 23% (74/319) 2024-06-26T07:19:05.5575174Z remote: Compressing objects: 24% (77/319) 2024-06-26T07:19:05.5575696Z remote: Compressing objects: 25% (80/319) 2024-06-26T07:19:05.5576216Z remote: Compressing objects: 26% (83/319) 2024-06-26T07:19:05.5576857Z remote: Compressing objects: 27% (87/319) 2024-06-26T07:19:05.5577364Z remote: Compressing objects: 28% (90/319) 2024-06-26T07:19:05.5577876Z remote: Compressing objects: 29% (93/319) 2024-06-26T07:19:05.5578399Z remote: Compressing objects: 30% (96/319) 2024-06-26T07:19:05.5579057Z remote: Compressing objects: 31% (99/319) 2024-06-26T07:19:05.5579708Z remote: Compressing objects: 32% (103/319) 2024-06-26T07:19:05.5580242Z remote: Compressing objects: 33% (106/319) 2024-06-26T07:19:05.5580913Z remote: Compressing objects: 34% (109/319) 2024-06-26T07:19:05.5581444Z remote: Compressing objects: 35% (112/319) 2024-06-26T07:19:05.5581962Z remote: Compressing objects: 36% (115/319) 2024-06-26T07:19:05.5582490Z remote: Compressing objects: 37% (119/319) 2024-06-26T07:19:05.5583165Z remote: Compressing objects: 38% (122/319) 2024-06-26T07:19:05.5584141Z remote: Compressing objects: 39% (125/319) 2024-06-26T07:19:05.5584871Z remote: Compressing objects: 40% (128/319) 2024-06-26T07:19:05.5585431Z remote: Compressing objects: 41% (131/319) 2024-06-26T07:19:05.5585952Z remote: Compressing objects: 42% (134/319) 2024-06-26T07:19:05.5586492Z remote: Compressing objects: 43% (138/319) 2024-06-26T07:19:05.5587110Z remote: Compressing objects: 44% (141/319) 2024-06-26T07:19:05.5587727Z remote: Compressing objects: 45% (144/319) 2024-06-26T07:19:05.5588254Z remote: Compressing objects: 46% (147/319) 2024-06-26T07:19:05.5588781Z remote: Compressing objects: 47% (150/319) 2024-06-26T07:19:05.5589293Z remote: Compressing objects: 48% (154/319) 2024-06-26T07:19:05.5589983Z remote: Compressing objects: 49% (157/319) 2024-06-26T07:19:05.5590513Z remote: Compressing objects: 50% (160/319) 2024-06-26T07:19:05.5591036Z remote: Compressing objects: 51% (163/319) 2024-06-26T07:19:05.5591553Z remote: Compressing objects: 52% (166/319) 2024-06-26T07:19:05.5592077Z remote: Compressing objects: 53% (170/319) 2024-06-26T07:19:05.5592758Z remote: Compressing objects: 54% (173/319) 2024-06-26T07:19:05.5593276Z remote: Compressing objects: 55% (176/319) 2024-06-26T07:19:05.5593803Z remote: Compressing objects: 56% (179/319) 2024-06-26T07:19:05.5594330Z remote: Compressing objects: 57% (182/319) 2024-06-26T07:19:05.5594855Z remote: Compressing objects: 58% (186/319) 2024-06-26T07:19:05.5595371Z remote: Compressing objects: 59% (189/319) 2024-06-26T07:19:05.5595893Z remote: Compressing objects: 60% (192/319) 2024-06-26T07:19:05.5596423Z remote: Compressing objects: 61% (195/319) 2024-06-26T07:19:05.5596984Z remote: Compressing objects: 62% (198/319) 2024-06-26T07:19:05.5597612Z remote: Compressing objects: 63% (201/319) 2024-06-26T07:19:05.5598138Z remote: Compressing objects: 64% (205/319) 2024-06-26T07:19:05.5598662Z remote: Compressing objects: 65% (208/319) 2024-06-26T07:19:05.5599359Z remote: Compressing objects: 66% (211/319) 2024-06-26T07:19:05.5599968Z remote: Compressing objects: 67% (214/319) 2024-06-26T07:19:05.5600491Z remote: Compressing objects: 68% (217/319) 2024-06-26T07:19:05.5601017Z remote: Compressing objects: 69% (221/319) 2024-06-26T07:19:05.5601539Z remote: Compressing objects: 70% (224/319) 2024-06-26T07:19:05.5602163Z remote: Compressing objects: 71% (227/319) 2024-06-26T07:19:05.5602690Z remote: Compressing objects: 72% (230/319) 2024-06-26T07:19:05.5603268Z remote: Compressing objects: 73% (233/319) 2024-06-26T07:19:05.5603910Z remote: Compressing objects: 74% (237/319) 2024-06-26T07:19:05.5604427Z remote: Compressing objects: 75% (240/319) 2024-06-26T07:19:05.5604951Z remote: Compressing objects: 76% (243/319) 2024-06-26T07:19:05.5605470Z remote: Compressing objects: 77% (246/319) 2024-06-26T07:19:05.5606130Z remote: Compressing objects: 78% (249/319) 2024-06-26T07:19:05.5606688Z remote: Compressing objects: 79% (253/319) 2024-06-26T07:19:05.5607290Z remote: Compressing objects: 80% (256/319) 2024-06-26T07:19:05.5607815Z remote: Compressing objects: 81% (259/319) 2024-06-26T07:19:05.5608354Z remote: Compressing objects: 82% (262/319) 2024-06-26T07:19:05.5609031Z remote: Compressing objects: 83% (265/319) 2024-06-26T07:19:05.5609748Z remote: Compressing objects: 84% (268/319) 2024-06-26T07:19:05.5610448Z remote: Compressing objects: 85% (272/319) 2024-06-26T07:19:05.5611117Z remote: Compressing objects: 86% (275/319) 2024-06-26T07:19:05.5611825Z remote: Compressing objects: 87% (278/319) 2024-06-26T07:19:05.5612704Z remote: Compressing objects: 88% (281/319) 2024-06-26T07:19:05.5613440Z remote: Compressing objects: 89% (284/319) 2024-06-26T07:19:05.5614170Z remote: Compressing objects: 90% (288/319) 2024-06-26T07:19:05.5614923Z remote: Compressing objects: 91% (291/319) 2024-06-26T07:19:05.5615690Z remote: Compressing objects: 92% (294/319) 2024-06-26T07:19:05.5616447Z remote: Compressing objects: 93% (297/319) 2024-06-26T07:19:05.5617156Z remote: Compressing objects: 94% (300/319) 2024-06-26T07:19:05.5617866Z remote: Compressing objects: 95% (304/319) 2024-06-26T07:19:05.5618591Z remote: Compressing objects: 96% (307/319) 2024-06-26T07:19:05.5619322Z remote: Compressing objects: 97% (310/319) 2024-06-26T07:19:05.5620032Z remote: Compressing objects: 98% (313/319) 2024-06-26T07:19:05.5620785Z remote: Compressing objects: 99% (316/319) 2024-06-26T07:19:05.5621533Z remote: Compressing objects: 100% (319/319) 2024-06-26T07:19:05.5622316Z remote: Compressing objects: 100% (319/319), done. 2024-06-26T07:19:05.5672826Z Receiving objects: 0% (1/30187) 2024-06-26T07:19:05.5710148Z Receiving objects: 1% (302/30187) 2024-06-26T07:19:05.5750035Z Receiving objects: 2% (604/30187) 2024-06-26T07:19:05.5785387Z Receiving objects: 3% (906/30187) 2024-06-26T07:19:05.5815551Z Receiving objects: 4% (1208/30187) 2024-06-26T07:19:05.5851575Z Receiving objects: 5% (1510/30187) 2024-06-26T07:19:05.5883369Z Receiving objects: 6% (1812/30187) 2024-06-26T07:19:05.5921875Z Receiving objects: 7% (2114/30187) 2024-06-26T07:19:05.5954300Z Receiving objects: 8% (2415/30187) 2024-06-26T07:19:05.5985329Z Receiving objects: 9% (2717/30187) 2024-06-26T07:19:05.6024140Z Receiving objects: 10% (3019/30187) 2024-06-26T07:19:05.6090405Z Receiving objects: 11% (3321/30187) 2024-06-26T07:19:05.6159223Z Receiving objects: 12% (3623/30187) 2024-06-26T07:19:05.6219443Z Receiving objects: 13% (3925/30187) 2024-06-26T07:19:05.6405338Z Receiving objects: 14% (4227/30187) 2024-06-26T07:19:05.6518508Z Receiving objects: 15% (4529/30187) 2024-06-26T07:19:05.6616463Z Receiving objects: 16% (4830/30187) 2024-06-26T07:19:05.6668716Z Receiving objects: 17% (5132/30187) 2024-06-26T07:19:05.6709986Z Receiving objects: 18% (5434/30187) 2024-06-26T07:19:05.6734988Z Receiving objects: 19% (5736/30187) 2024-06-26T07:19:05.6798433Z Receiving objects: 20% (6038/30187) 2024-06-26T07:19:05.6817889Z Receiving objects: 21% (6340/30187) 2024-06-26T07:19:05.6838021Z Receiving objects: 22% (6642/30187) 2024-06-26T07:19:05.7890726Z Receiving objects: 23% (6944/30187) 2024-06-26T07:19:06.5626943Z Receiving objects: 24% (7245/30187) 2024-06-26T07:19:06.6638962Z Receiving objects: 24% (7431/30187), 80.41 MiB | 80.48 MiB/s 2024-06-26T07:19:07.1477930Z Receiving objects: 25% (7547/30187), 80.41 MiB | 80.48 MiB/s 2024-06-26T07:19:07.2141202Z Receiving objects: 26% (7849/30187), 118.97 MiB | 79.36 MiB/s 2024-06-26T07:19:07.2807851Z Receiving objects: 27% (8151/30187), 118.97 MiB | 79.36 MiB/s 2024-06-26T07:19:07.3462002Z Receiving objects: 28% (8453/30187), 118.97 MiB | 79.36 MiB/s 2024-06-26T07:19:07.4123296Z Receiving objects: 29% (8755/30187), 118.97 MiB | 79.36 MiB/s 2024-06-26T07:19:07.4785297Z Receiving objects: 30% (9057/30187), 118.97 MiB | 79.36 MiB/s 2024-06-26T07:19:07.5445181Z Receiving objects: 31% (9358/30187), 118.97 MiB | 79.36 MiB/s 2024-06-26T07:19:07.5626512Z Receiving objects: 32% (9660/30187), 118.97 MiB | 79.36 MiB/s 2024-06-26T07:19:07.6110179Z Receiving objects: 32% (9742/30187), 161.14 MiB | 80.61 MiB/s 2024-06-26T07:19:07.8324384Z Receiving objects: 33% (9962/30187), 161.14 MiB | 80.61 MiB/s 2024-06-26T07:19:07.9047182Z Receiving objects: 34% (10264/30187), 161.14 MiB | 80.61 MiB/s 2024-06-26T07:19:07.9168797Z Receiving objects: 35% (10566/30187), 161.14 MiB | 80.61 MiB/s 2024-06-26T07:19:08.1283559Z Receiving objects: 36% (10868/30187), 161.14 MiB | 80.61 MiB/s 2024-06-26T07:19:08.3811674Z Receiving objects: 37% (11170/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4013489Z Receiving objects: 38% (11472/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4027573Z Receiving objects: 39% (11773/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4051772Z Receiving objects: 40% (12075/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4064826Z Receiving objects: 41% (12377/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4078236Z Receiving objects: 42% (12679/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4089130Z Receiving objects: 43% (12981/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4098365Z Receiving objects: 44% (13283/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4117761Z Receiving objects: 45% (13585/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4128916Z Receiving objects: 46% (13887/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4133271Z Receiving objects: 47% (14188/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4139197Z Receiving objects: 48% (14490/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4160426Z Receiving objects: 49% (14792/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4176514Z Receiving objects: 50% (15094/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4187104Z Receiving objects: 51% (15396/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4381299Z Receiving objects: 52% (15698/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4385897Z Receiving objects: 53% (16000/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4393056Z Receiving objects: 54% (16301/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4399043Z Receiving objects: 55% (16603/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4404576Z Receiving objects: 56% (16905/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4420772Z Receiving objects: 57% (17207/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4430809Z Receiving objects: 58% (17509/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4439297Z Receiving objects: 59% (17811/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4452440Z Receiving objects: 60% (18113/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4460879Z Receiving objects: 61% (18415/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4470601Z Receiving objects: 62% (18716/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4622452Z Receiving objects: 63% (19018/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4629050Z Receiving objects: 64% (19320/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4635523Z Receiving objects: 65% (19622/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4642395Z Receiving objects: 66% (19924/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4652091Z Receiving objects: 67% (20226/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4660723Z Receiving objects: 68% (20528/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.4928567Z Receiving objects: 69% (20830/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5343412Z Receiving objects: 70% (21131/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5349817Z Receiving objects: 71% (21433/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5362368Z Receiving objects: 72% (21735/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5376151Z Receiving objects: 73% (22037/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5389989Z Receiving objects: 74% (22339/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5400521Z Receiving objects: 75% (22641/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5410406Z Receiving objects: 76% (22943/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5429742Z Receiving objects: 77% (23244/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5436827Z Receiving objects: 78% (23546/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5446005Z Receiving objects: 79% (23848/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5469573Z Receiving objects: 80% (24150/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.5626454Z Receiving objects: 81% (24452/30187), 192.75 MiB | 77.16 MiB/s 2024-06-26T07:19:08.7575271Z Receiving objects: 81% (24547/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.7783803Z Receiving objects: 82% (24754/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.7815297Z Receiving objects: 83% (25056/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.7825529Z Receiving objects: 84% (25358/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.7850987Z Receiving objects: 85% (25659/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.7865811Z Receiving objects: 86% (25961/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.7894068Z Receiving objects: 87% (26263/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8021691Z Receiving objects: 88% (26565/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8027776Z Receiving objects: 89% (26867/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8035372Z Receiving objects: 90% (27169/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8040662Z Receiving objects: 91% (27471/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8047994Z Receiving objects: 92% (27773/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8055073Z Receiving objects: 93% (28074/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8172183Z Receiving objects: 94% (28376/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8189199Z Receiving objects: 95% (28678/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8199802Z Receiving objects: 96% (28980/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8219696Z Receiving objects: 97% (29282/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8234213Z Receiving objects: 98% (29584/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8273569Z Receiving objects: 99% (29886/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8274470Z remote: Total 30187 (delta 255), reused 486 (delta 215), pack-reused 29641 2024-06-26T07:19:08.8291646Z Receiving objects: 100% (30187/30187), 229.92 MiB | 76.71 MiB/s 2024-06-26T07:19:08.8292317Z Receiving objects: 100% (30187/30187), 245.25 MiB | 75.08 MiB/s, done. 2024-06-26T07:19:08.8318294Z Resolving deltas: 0% (0/16212) 2024-06-26T07:19:08.8331875Z Resolving deltas: 1% (163/16212) 2024-06-26T07:19:08.8341460Z Resolving deltas: 2% (405/16212) 2024-06-26T07:19:08.8360054Z Resolving deltas: 3% (503/16212) 2024-06-26T07:19:08.8379727Z Resolving deltas: 4% (649/16212) 2024-06-26T07:19:08.8388509Z Resolving deltas: 5% (901/16212) 2024-06-26T07:19:08.8396906Z Resolving deltas: 6% (989/16212) 2024-06-26T07:19:08.8406968Z Resolving deltas: 7% (1140/16212) 2024-06-26T07:19:08.8415578Z Resolving deltas: 8% (1297/16212) 2024-06-26T07:19:08.8424823Z Resolving deltas: 9% (1463/16212) 2024-06-26T07:19:08.8435856Z Resolving deltas: 10% (1627/16212) 2024-06-26T07:19:08.8446431Z Resolving deltas: 11% (1790/16212) 2024-06-26T07:19:08.8455395Z Resolving deltas: 12% (1949/16212) 2024-06-26T07:19:08.8468529Z Resolving deltas: 13% (2108/16212) 2024-06-26T07:19:08.8478320Z Resolving deltas: 14% (2270/16212) 2024-06-26T07:19:08.8495034Z Resolving deltas: 15% (2436/16212) 2024-06-26T07:19:08.8511830Z Resolving deltas: 16% (2612/16212) 2024-06-26T07:19:08.8526919Z Resolving deltas: 17% (2759/16212) 2024-06-26T07:19:08.8568029Z Resolving deltas: 18% (2920/16212) 2024-06-26T07:19:08.8585489Z Resolving deltas: 19% (3081/16212) 2024-06-26T07:19:08.8601893Z Resolving deltas: 20% (3249/16212) 2024-06-26T07:19:08.8604748Z Resolving deltas: 21% (3523/16212) 2024-06-26T07:19:08.8612410Z Resolving deltas: 22% (3575/16212) 2024-06-26T07:19:08.8627711Z Resolving deltas: 23% (3732/16212) 2024-06-26T07:19:08.8630301Z Resolving deltas: 24% (4014/16212) 2024-06-26T07:19:08.8642730Z Resolving deltas: 25% (4068/16212) 2024-06-26T07:19:08.8676650Z Resolving deltas: 26% (4218/16212) 2024-06-26T07:19:08.8697505Z Resolving deltas: 27% (4379/16212) 2024-06-26T07:19:08.8710660Z Resolving deltas: 28% (4541/16212) 2024-06-26T07:19:08.8725162Z Resolving deltas: 29% (4710/16212) 2024-06-26T07:19:08.8738708Z Resolving deltas: 30% (4884/16212) 2024-06-26T07:19:08.8754072Z Resolving deltas: 31% (5027/16212) 2024-06-26T07:19:08.8762431Z Resolving deltas: 32% (5195/16212) 2024-06-26T07:19:08.8772997Z Resolving deltas: 33% (5355/16212) 2024-06-26T07:19:08.8783607Z Resolving deltas: 34% (5517/16212) 2024-06-26T07:19:08.8803936Z Resolving deltas: 35% (5682/16212) 2024-06-26T07:19:08.8820273Z Resolving deltas: 36% (5838/16212) 2024-06-26T07:19:08.8836557Z Resolving deltas: 37% (6000/16212) 2024-06-26T07:19:08.8849429Z Resolving deltas: 38% (6187/16212) 2024-06-26T07:19:08.8864873Z Resolving deltas: 39% (6323/16212) 2024-06-26T07:19:08.8874288Z Resolving deltas: 40% (6485/16212) 2024-06-26T07:19:08.8890774Z Resolving deltas: 41% (6648/16212) 2024-06-26T07:19:08.8901578Z Resolving deltas: 42% (6845/16212) 2024-06-26T07:19:08.8909262Z Resolving deltas: 43% (6973/16212) 2024-06-26T07:19:08.8918561Z Resolving deltas: 44% (7177/16212) 2024-06-26T07:19:08.8931299Z Resolving deltas: 45% (7316/16212) 2024-06-26T07:19:08.8955152Z Resolving deltas: 46% (7459/16212) 2024-06-26T07:19:08.8966539Z Resolving deltas: 47% (7620/16212) 2024-06-26T07:19:08.8985371Z Resolving deltas: 48% (7785/16212) 2024-06-26T07:19:08.8997840Z Resolving deltas: 49% (7944/16212) 2024-06-26T07:19:08.9008489Z Resolving deltas: 50% (8111/16212) 2024-06-26T07:19:08.9015151Z Resolving deltas: 51% (8284/16212) 2024-06-26T07:19:08.9024948Z Resolving deltas: 52% (8440/16212) 2024-06-26T07:19:08.9043270Z Resolving deltas: 53% (8595/16212) 2024-06-26T07:19:08.9047238Z Resolving deltas: 54% (8839/16212) 2024-06-26T07:19:08.9053722Z Resolving deltas: 55% (8921/16212) 2024-06-26T07:19:08.9066247Z Resolving deltas: 56% (9081/16212) 2024-06-26T07:19:08.9076236Z Resolving deltas: 57% (9246/16212) 2024-06-26T07:19:08.9083630Z Resolving deltas: 58% (9406/16212) 2024-06-26T07:19:08.9093461Z Resolving deltas: 59% (9592/16212) 2024-06-26T07:19:08.9099728Z Resolving deltas: 60% (9863/16212) 2024-06-26T07:19:08.9103729Z Resolving deltas: 61% (9980/16212) 2024-06-26T07:19:08.9112708Z Resolving deltas: 62% (10063/16212) 2024-06-26T07:19:08.9123076Z Resolving deltas: 63% (10240/16212) 2024-06-26T07:19:08.9127356Z Resolving deltas: 64% (10454/16212) 2024-06-26T07:19:08.9141653Z Resolving deltas: 65% (10552/16212) 2024-06-26T07:19:08.9162544Z Resolving deltas: 66% (10844/16212) 2024-06-26T07:19:08.9171083Z Resolving deltas: 68% (11151/16212) 2024-06-26T07:19:08.9187019Z Resolving deltas: 69% (11277/16212) 2024-06-26T07:19:08.9195887Z Resolving deltas: 70% (11350/16212) 2024-06-26T07:19:08.9205455Z Resolving deltas: 71% (11511/16212) 2024-06-26T07:19:08.9218317Z Resolving deltas: 72% (11673/16212) 2024-06-26T07:19:08.9231200Z Resolving deltas: 73% (11863/16212) 2024-06-26T07:19:08.9244894Z Resolving deltas: 74% (11997/16212) 2024-06-26T07:19:08.9277910Z Resolving deltas: 75% (12184/16212) 2024-06-26T07:19:08.9281855Z Resolving deltas: 78% (12724/16212) 2024-06-26T07:19:08.9290683Z Resolving deltas: 79% (12810/16212) 2024-06-26T07:19:08.9297821Z Resolving deltas: 80% (12988/16212) 2024-06-26T07:19:08.9307462Z Resolving deltas: 81% (13142/16212) 2024-06-26T07:19:08.9314920Z Resolving deltas: 82% (13302/16212) 2024-06-26T07:19:08.9321058Z Resolving deltas: 83% (13457/16212) 2024-06-26T07:19:08.9331617Z Resolving deltas: 84% (13625/16212) 2024-06-26T07:19:08.9351341Z Resolving deltas: 85% (13781/16212) 2024-06-26T07:19:08.9360535Z Resolving deltas: 86% (13944/16212) 2024-06-26T07:19:08.9367850Z Resolving deltas: 87% (14109/16212) 2024-06-26T07:19:08.9375096Z Resolving deltas: 88% (14281/16212) 2024-06-26T07:19:08.9383593Z Resolving deltas: 89% (14443/16212) 2024-06-26T07:19:08.9392563Z Resolving deltas: 90% (14611/16212) 2024-06-26T07:19:08.9398981Z Resolving deltas: 91% (14774/16212) 2024-06-26T07:19:08.9408810Z Resolving deltas: 92% (14921/16212) 2024-06-26T07:19:08.9412887Z Resolving deltas: 93% (15142/16212) 2024-06-26T07:19:08.9420309Z Resolving deltas: 94% (15263/16212) 2024-06-26T07:19:08.9428002Z Resolving deltas: 95% (15422/16212) 2024-06-26T07:19:08.9434813Z Resolving deltas: 96% (15569/16212) 2024-06-26T07:19:08.9441426Z Resolving deltas: 97% (15726/16212) 2024-06-26T07:19:08.9450897Z Resolving deltas: 98% (15895/16212) 2024-06-26T07:19:08.9484981Z Resolving deltas: 99% (16051/16212) 2024-06-26T07:19:08.9485401Z Resolving deltas: 100% (16212/16212) 2024-06-26T07:19:08.9485827Z Resolving deltas: 100% (16212/16212), done. 2024-06-26T07:19:10.0046808Z + pushd torchbench 2024-06-26T07:19:10.0047590Z ~/workspace/torchbench ~/workspace 2024-06-26T07:19:10.0048579Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:19:10.0370459Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:19:10.0371032Z 2024-06-26T07:19:10.0371458Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:19:10.0372499Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:19:10.0373344Z state without impacting any branches by switching back to a branch. 2024-06-26T07:19:10.0373818Z 2024-06-26T07:19:10.0374126Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:19:10.0374934Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:19:10.0375354Z 2024-06-26T07:19:10.0375556Z git switch -c 2024-06-26T07:19:10.0375826Z 2024-06-26T07:19:10.0375971Z Or undo this operation with: 2024-06-26T07:19:10.0376218Z 2024-06-26T07:19:10.0376350Z git switch - 2024-06-26T07:19:10.0376539Z 2024-06-26T07:19:10.0376872Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:19:10.0377368Z 2024-06-26T07:19:10.0377535Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:19:10.0377986Z + '[' '' ']' 2024-06-26T07:19:10.0378341Z + python install.py --continue_on_fail 2024-06-26T07:19:12.0060031Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:19:27.6640476Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:19:30.0165433Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:19:32.6242778Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:19:34.9708537Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:19:34.9858145Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:19:43.7244968Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:19:47.0903799Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:19:50.4405992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:19:53.7960212Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:19:53.7962535Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:19:55.6127465Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:19:59.1228047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:19:59.1379742Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:20:55.9966509Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:21:05.2652339Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:21:12.9329787Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:21:20.2974776Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:21:29.4739395Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:21:37.1953779Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:21:44.1691822Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:21:51.7051432Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:21:59.5393585Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:22:07.3449272Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:22:14.7700752Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:22:22.2809026Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:22:26.3378512Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:23:05.3048233Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:23:11.2965718Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:23:17.9123768Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:23:28.9157343Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:23:31.1144479Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:23:33.5965426Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:23:41.5392218Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:23:49.5774958Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:23:56.2481550Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:24:04.8867419Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:24:12.0350407Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:24:18.3056082Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:24:25.7945209Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:24:42.0462029Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:24:49.3799820Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:24:54.7083489Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:25:00.7281694Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:25:08.5390133Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:25:08.5392867Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:25:21.8696146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:25:27.1684225Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:25:27.1685896Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:25:35.7361074Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:25:38.0297136Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:25:40.3134957Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:26:20.0795419Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:27:39.7815188Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:27:39.7971182Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:27:42.3499101Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:27:42.3654992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:27:42.3811212Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:27:42.3963637Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:27:42.4115647Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:27:42.4265978Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:28:02.8028961Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:28:02.8032040Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:28:05.1116981Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:28:08.4824647Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:28:08.4975247Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:28:08.5132110Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:28:08.5286221Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:28:08.5439202Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:28:08.5591858Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:28:15.8138883Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:28:18.3361928Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:28:22.9017117Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:28:22.9173417Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:28:22.9332891Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:28:22.9485368Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:28:22.9637779Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:28:22.9791956Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:28:34.6128095Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:28:45.7897533Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:28:45.8054695Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:28:45.8058504Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:28:45.8060087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:28:49.2253734Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:28:52.0603349Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:28:52.0768370Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:29:13.9095997Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:29:20.6934600Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:29:26.2165361Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:29:33.4991666Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:29:33.5146814Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:29:33.5295448Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:29:33.5452200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:29:33.5603512Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:29:33.5756641Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:29:33.5911126Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:29:33.6063833Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:29:45.0974205Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:29:58.0287399Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:29:58.0442747Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:30:00.9932238Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:30:04.2247112Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:30:04.6476804Z + popd 2024-06-26T07:30:04.6478403Z ~/workspace 2024-06-26T07:30:04.6478818Z + [[ aot_eager_torchbench != *cpu_inductor* ]] 2024-06-26T07:30:04.6479290Z + install_torchrec_and_fbgemm 2024-06-26T07:30:04.6479660Z + local torchrec_commit 2024-06-26T07:30:04.6480014Z ++ get_pinned_commit torchrec 2024-06-26T07:30:04.6480437Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:30:04.6567500Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:30:04.6568158Z + local fbgemm_commit 2024-06-26T07:30:04.6569161Z ++ get_pinned_commit fbgemm 2024-06-26T07:30:04.6569557Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:30:04.6576996Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:04.6577711Z + pip_uninstall torchrec-nightly 2024-06-26T07:30:04.6578166Z + pip uninstall -y torchrec-nightly 2024-06-26T07:30:05.0174147Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:30:05.0597973Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:30:05.0598519Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:30:05.4070338Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:30:05.4532036Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:05.4533110Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:30:05.8927371Z Collecting setuptools-git-versioning 2024-06-26T07:30:05.9100100Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:30:06.0098313Z Collecting scikit-build 2024-06-26T07:30:06.0131049Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:30:06.1146898Z Collecting pyre-extensions 2024-06-26T07:30:06.1195108Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:30:06.1618923Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:30:06.1621876Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:30:06.2040933Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:30:06.2075287Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:30:06.2691005Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:30:06.2701913Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:30:06.2707652Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:30:06.3241275Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:30:06.3273602Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:30:06.3695794Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:30:06.3918007Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:30:06.3965301Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:30:06.4529217Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:30:06.5091696Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:30:06.5705558Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:30:06.6217095Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:30:08.7880235Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:30:09.1075958Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:30:09.2238593Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:30:09.2239768Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:09.2241651Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:30:09.5653239Z Collecting fbgemm-gpu 2024-06-26T07:30:09.5655546Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-loqc9n9s/fbgemm-gpu_33613fb630aa4bfeaebcee8452b08ba1 2024-06-26T07:30:09.5670783Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-loqc9n9s/fbgemm-gpu_33613fb630aa4bfeaebcee8452b08ba1 2024-06-26T07:30:10.4989296Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:30:10.5003584Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:10.9058648Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:11.3585261Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:30:11.3586287Z Running command git submodule update --init --recursive -q 2024-06-26T07:30:19.3041096Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:30:19.3061942Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:30:19.3068557Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:12:28.5235441Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T08:12:28.7036293Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252630317 sha256=ff8429673c9cb4967ddfc6bea3d598ebdd62cc25ce642dd184de6cf9b78b5b76 2024-06-26T08:12:28.7038276Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:12:28.7056471Z Successfully built fbgemm-gpu 2024-06-26T08:12:30.8595590Z Installing collected packages: fbgemm-gpu 2024-06-26T08:12:34.0056506Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:12:34.6368592Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:34.6370111Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:34.9746506Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:34.9749794Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-mo2lv9ah 2024-06-26T08:12:34.9763938Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-mo2lv9ah 2024-06-26T08:12:35.6371525Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:12:35.6389346Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:35.9115309Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:36.2503428Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:12:36.4772701Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:12:36.4821485Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:12:36.4825004Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:12:36.4829558Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:12:36.5349584Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:12:36.5522610Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:12:36.5583154Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:12:36.5586707Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:12:36.5600476Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:12:36.5621386Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:12:36.6283452Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:12:36.6287693Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:12:36.6292611Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:12:36.7050795Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:12:36.7056168Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:12:36.7241358Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:12:36.7288219Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:12:36.7452773Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:12:36.7455873Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:12:36.7490171Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:12:36.7560399Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:12:36.7583818Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:12:36.7586617Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:12:36.7589364Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:12:36.7592133Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:12:36.8245627Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:12:36.8550234Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:12:36.8690838Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:12:36.8873606Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:12:36.8964374Z Building wheels for collected packages: torchrec 2024-06-26T08:12:37.3378298Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:12:37.3387753Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=20b996febb6d6ab16bbfca6342a666543dcfd821bc320703c4f5e9645d2d271d 2024-06-26T08:12:37.3389383Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:12:37.3411664Z Successfully built torchrec 2024-06-26T08:12:39.4153488Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:12:39.9836698Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:12:40.1168747Z ++ pwd 2024-06-26T08:12:40.1170973Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:12:40.1171532Z + test_dynamo_benchmark torchbench 1 2024-06-26T08:12:40.1174305Z ++ pwd 2024-06-26T08:12:40.1176134Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:40.1176882Z + local suite=torchbench 2024-06-26T08:12:40.1177276Z + shift 2024-06-26T08:12:40.1177629Z + local shard_id=1 2024-06-26T08:12:40.1178071Z + shift 2024-06-26T08:12:40.1178425Z + [[ aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:12:40.1179030Z + [[ aot_eager_torchbench == *perf* ]] 2024-06-26T08:12:40.1179672Z + [[ aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T08:12:40.1180302Z + [[ aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:12:40.1181828Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-06-26T08:12:40.1182457Z ++ pwd 2024-06-26T08:12:40.1182927Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:40.1183798Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:12:40.1210192Z + local name=inference 2024-06-26T08:12:40.1210652Z + shift 2024-06-26T08:12:40.1211079Z + local suite=torchbench 2024-06-26T08:12:40.1211541Z + shift 2024-06-26T08:12:40.1211895Z + local shard_id=1 2024-06-26T08:12:40.1212540Z + shift 2024-06-26T08:12:40.1212913Z + partition_flags=() 2024-06-26T08:12:40.1213282Z + local partition_flags 2024-06-26T08:12:40.1213665Z + [[ -n 2 ]] 2024-06-26T08:12:40.1213972Z + [[ -n 1 ]] 2024-06-26T08:12:40.1217233Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:12:40.1218008Z + [[ aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:12:40.1218479Z + [[ aot_eager_torchbench == *perf* ]] 2024-06-26T08:12:40.1218925Z + [[ aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:12:40.1220542Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:12:45.7348813Z 2024-06-26T08:12:46.2241910Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:46.2242373Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:46.2279404Z cuda eval llama 2024-06-26T08:12:52.6534284Z pass 2024-06-26T08:12:52.6694900Z TIMING: backend_compile:3.87426 entire_frame_compile:5.86492 2024-06-26T08:12:52.6695994Z STATS: call_* op count: 1058 | FakeTensor.__torch_dispatch__:365 | FakeTensorMode.__torch_dispatch__:22394 | ProxyTorchDispatchMode.__torch_dispatch__:8227 2024-06-26T08:12:52.6697109Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique) 2024-06-26T08:12:56.0764188Z 2024-06-26T08:12:56.3468926Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:56.3469620Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:56.3470684Z cuda eval llama_v2_7b_16h 2024-06-26T08:12:56.3476096Z Traceback (most recent call last): 2024-06-26T08:12:56.3477005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:12:56.3477726Z ) = runner.load_model( 2024-06-26T08:12:56.3478412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:12:56.3479135Z benchmark = benchmark_cls( 2024-06-26T08:12:56.3479903Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:12:56.3481051Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:12:56.3482408Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-06-26T08:12:56.3484119Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:12:56.3486072Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:12:56.3487832Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:12:56.3489277Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:12:56.3490483Z self._skip_by_device_name() 2024-06-26T08:12:56.3491720Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:12:56.3493610Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:12:56.3495371Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-06-26T08:12:56.3496299Z 2024-06-26T08:12:56.3496479Z model_fail_to_load 2024-06-26T08:12:59.4912428Z 2024-06-26T08:12:59.7608843Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:59.7609400Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:59.7609895Z cuda eval llava 2024-06-26T08:12:59.7614023Z Traceback (most recent call last): 2024-06-26T08:12:59.7614965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:12:59.7615861Z ) = runner.load_model( 2024-06-26T08:12:59.7617048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:12:59.7617944Z benchmark = benchmark_cls( 2024-06-26T08:12:59.7618717Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:12:59.7619516Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:12:59.7620357Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-06-26T08:12:59.7621445Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:12:59.7622694Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:12:59.7624047Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:12:59.7625942Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:12:59.7626762Z self._skip_by_device_name() 2024-06-26T08:12:59.7627559Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:12:59.7628779Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:12:59.7629852Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-06-26T08:12:59.7630411Z 2024-06-26T08:12:59.7630536Z model_fail_to_load 2024-06-26T08:13:02.9131746Z 2024-06-26T08:13:03.4619665Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:03.4620179Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:03.4627041Z cuda eval maml 2024-06-26T08:13:03.4630098Z pass_due_to_skip 2024-06-26T08:13:03.4638328Z TIMING: 2024-06-26T08:13:03.4639068Z STATS: call_* op count: 0 2024-06-26T08:13:03.4640037Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:06.6422083Z 2024-06-26T08:13:07.1110850Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:07.1111567Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:07.1117972Z cuda eval maml_omniglot 2024-06-26T08:13:07.7332782Z pass 2024-06-26T08:13:07.7333363Z TIMING: backend_compile:0.27932 entire_frame_compile:0.4496 2024-06-26T08:13:07.7334797Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:89 | FakeTensorMode.__torch_dispatch__:1572 | ProxyTorchDispatchMode.__torch_dispatch__:340 2024-06-26T08:13:07.7336376Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:11.0457715Z 2024-06-26T08:13:11.2414565Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-06-26T08:13:11.2609663Z 2024-06-26T08:13:11.2609999Z 2024-06-26T08:13:11.3134808Z 0% 0.00/16.9M [00:00 will be ignored 2024-06-26T08:13:53.4360718Z [rank0]:W0626 08:13:53.434000 140085303042688 torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:13:53.4549469Z [rank0]:W0626 08:13:53.454000 140085303042688 torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:13:53.4554145Z [rank0]:W0626 08:13:53.454000 140085303042688 torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:13:53.4555689Z [rank0]:W0626 08:13:53.454000 140085303042688 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:13:53.4557245Z [rank0]:W0626 08:13:53.454000 140085303042688 torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:13:53.4558708Z [rank0]:W0626 08:13:53.454000 140085303042688 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ 0 │ 157 │ 44910720 │ 2024-06-26T08:13:53.4560423Z [rank0]:W0626 08:13:53.454000 140085303042688 torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:14:01.0383676Z [rank0]:W0626 08:14:01.037000 140085303042688 torch/_dynamo/variables/tensor.py:714] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:14:01.0385287Z [rank0]:W0626 08:14:01.037000 140085303042688 torch/_dynamo/variables/tensor.py:714] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:14:01.0386435Z [rank0]:W0626 08:14:01.037000 140085303042688 torch/_dynamo/variables/tensor.py:714] [5/0] or: 2024-06-26T08:14:01.0387586Z [rank0]:W0626 08:14:01.037000 140085303042688 torch/_dynamo/variables/tensor.py:714] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:14:01.0388917Z [rank0]:W0626 08:14:01.037000 140085303042688 torch/_dynamo/variables/tensor.py:714] [5/0] to include these operations in the captured graph. 2024-06-26T08:14:01.0390034Z [rank0]:W0626 08:14:01.037000 140085303042688 torch/_dynamo/variables/tensor.py:714] [5/0] 2024-06-26T08:14:04.2227491Z pass 2024-06-26T08:14:04.2244175Z TIMING: entire_frame_compile:18.03813 backend_compile:8.63069 2024-06-26T08:14:04.2245279Z STATS: call_* op count: 877 | FakeTensor.__torch_dispatch__:2512 | FakeTensorMode.__torch_dispatch__:41738 | ProxyTorchDispatchMode.__torch_dispatch__:9290 2024-06-26T08:14:04.2246393Z Dynamo produced 7 graphs covering 877 ops with 5 graph breaks (3 unique) 2024-06-26T08:14:04.6552617Z [rank0]:[W626 08:14:04.692266419 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:14:08.2769743Z 2024-06-26T08:14:08.5467942Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:08.5468959Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:08.5469880Z cuda eval moondream 2024-06-26T08:14:08.5474425Z Traceback (most recent call last): 2024-06-26T08:14:08.5475882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:14:08.5477029Z ) = runner.load_model( 2024-06-26T08:14:08.5477703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:14:08.5478515Z benchmark = benchmark_cls( 2024-06-26T08:14:08.5479255Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:14:08.5480044Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:14:08.5481027Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-06-26T08:14:08.5482444Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:14:08.5483710Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:14:08.5484882Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:14:08.5485845Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:14:08.5486622Z self._skip_by_device_name() 2024-06-26T08:14:08.5487446Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:14:08.5488673Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:14:08.5489754Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-06-26T08:14:08.5490336Z 2024-06-26T08:14:08.5490713Z model_fail_to_load 2024-06-26T08:14:11.6661495Z 2024-06-26T08:14:13.5379375Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:14:13.7699991Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:14:13.7701833Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:14:13.7705507Z using fused AdamW: True 2024-06-26T08:14:14.0262885Z 2024-06-26T08:14:14.0263578Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:14:14.0302739Z cuda eval nanogpt 2024-06-26T08:14:19.4827528Z pass 2024-06-26T08:14:19.4828148Z TIMING: backend_compile:2.97009 entire_frame_compile:4.91719 2024-06-26T08:14:19.4829218Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:18097 | FakeTensor.__torch_dispatch__:819 | ProxyTorchDispatchMode.__torch_dispatch__:6111 2024-06-26T08:14:19.4830320Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:22.8256717Z 2024-06-26T08:14:25.1568742Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:25.1569250Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:14:25.1572833Z cuda eval nvidia_deeprecommender 2024-06-26T08:14:25.8121959Z pass 2024-06-26T08:14:25.8123413Z TIMING: backend_compile:0.19175 entire_frame_compile:0.38092 2024-06-26T08:14:25.8124414Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:815 | ProxyTorchDispatchMode.__torch_dispatch__:242 2024-06-26T08:14:25.8125482Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:29.0707298Z 2024-06-26T08:14:29.8666892Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:29.8667793Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:29.8685086Z cuda eval opacus_cifar10 2024-06-26T08:14:34.3356525Z pass 2024-06-26T08:14:34.3357092Z TIMING: backend_compile:1.09057 entire_frame_compile:4.14543 2024-06-26T08:14:34.3358108Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:3450 | ProxyTorchDispatchMode.__torch_dispatch__:1188 2024-06-26T08:14:34.3359161Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:37.6770063Z 2024-06-26T08:14:38.2373836Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:38.2374395Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:38.2439558Z cuda eval phlippe_densenet 2024-06-26T08:14:43.2923233Z pass 2024-06-26T08:14:43.2925643Z TIMING: backend_compile:2.83119 entire_frame_compile:4.69676 2024-06-26T08:14:43.2926849Z STATS: call_* op count: 372 | FakeTensor.__torch_dispatch__:993 | FakeTensorMode.__torch_dispatch__:21942 | ProxyTorchDispatchMode.__torch_dispatch__:4641 2024-06-26T08:14:43.2928139Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:46.6703204Z 2024-06-26T08:14:47.1491965Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:47.1492458Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:47.1518222Z cuda eval phlippe_resnet 2024-06-26T08:14:49.1853238Z pass 2024-06-26T08:14:49.1853767Z TIMING: backend_compile:1.14744 entire_frame_compile:1.78625 2024-06-26T08:14:49.1854824Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:391 | FakeTensorMode.__torch_dispatch__:8229 | ProxyTorchDispatchMode.__torch_dispatch__:1757 2024-06-26T08:14:49.1855936Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:52.4943540Z 2024-06-26T08:14:52.8159368Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:52.8160042Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:52.8160506Z cuda eval pyhpc_equation_of_state 2024-06-26T08:14:54.3704960Z pass 2024-06-26T08:14:54.3705569Z TIMING: backend_compile:0.9878 entire_frame_compile:1.38168 2024-06-26T08:14:54.3706477Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:8987 | ProxyTorchDispatchMode.__torch_dispatch__:3732 2024-06-26T08:14:54.3707427Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:57.6237134Z 2024-06-26T08:14:57.9337632Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:57.9338501Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:57.9342377Z cuda eval pyhpc_isoneutral_mixing 2024-06-26T08:15:07.2438000Z pass 2024-06-26T08:15:07.2438597Z TIMING: backend_compile:7.6034 entire_frame_compile:9.03809 2024-06-26T08:15:07.2439691Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:37150 | FakeTensor.__torch_dispatch__:18 | ProxyTorchDispatchMode.__torch_dispatch__:13834 2024-06-26T08:15:07.2440788Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:10.6314080Z 2024-06-26T08:15:11.3673018Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:15:11.4480837Z 2024-06-26T08:15:11.4481392Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:11.4482210Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:15:11.4936547Z cuda eval pyhpc_turbulent_kinetic_energy 2024-06-26T08:15:11.4937384Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:15:21.6135232Z pass 2024-06-26T08:15:21.6233321Z TIMING: backend_compile:8.25189 entire_frame_compile:9.83322 2024-06-26T08:15:21.6234596Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:39778 | FakeTensor.__torch_dispatch__:71 | ProxyTorchDispatchMode.__torch_dispatch__:14474 2024-06-26T08:15:21.6235698Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:25.0576774Z 2024-06-26T08:15:25.7318204Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:25.7318675Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:25.7330709Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-06-26T08:15:33.1384484Z pass 2024-06-26T08:15:33.1385483Z TIMING: backend_compile:1.20207 entire_frame_compile:1.76676 2024-06-26T08:15:33.1388828Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:7533 | FakeTensor.__torch_dispatch__:297 | ProxyTorchDispatchMode.__torch_dispatch__:2092 2024-06-26T08:15:33.1390187Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:36.4602241Z 2024-06-26T08:15:37.4998415Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:37.4998888Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:15:37.5019924Z cuda eval pytorch_stargan 2024-06-26T08:15:54.8151192Z pass 2024-06-26T08:15:54.8195296Z TIMING: backend_compile:2.04085 entire_frame_compile:2.65401 2024-06-26T08:15:54.8196549Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:12793 | FakeTensor.__torch_dispatch__:329 | ProxyTorchDispatchMode.__torch_dispatch__:3694 2024-06-26T08:15:54.8197727Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:58.1688907Z 2024-06-26T08:15:58.7666349Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:58.7666895Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:58.7688468Z cuda eval pytorch_unet 2024-06-26T08:16:08.6833478Z pass 2024-06-26T08:16:08.6995416Z TIMING: backend_compile:1.60881 entire_frame_compile:2.46029 2024-06-26T08:16:08.6996507Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:462 | FakeTensorMode.__torch_dispatch__:11387 | ProxyTorchDispatchMode.__torch_dispatch__:3036 2024-06-26T08:16:08.6997616Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:16:11.9698637Z 2024-06-26T08:16:12.7315272Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-06-26T08:16:12.7530460Z 2024-06-26T08:16:12.7530552Z 2024-06-26T08:16:12.8533011Z 0% 0.00/230M [00:00 will be ignored 2024-06-26T08:30:13.4419275Z pass 2024-06-26T08:30:13.4420036Z TIMING: entire_frame_compile:2.27263 backend_compile:1.4356 2024-06-26T08:30:13.4421107Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:398 | FakeTensorMode.__torch_dispatch__:4179 | ProxyTorchDispatchMode.__torch_dispatch__:1170 2024-06-26T08:30:13.4422208Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:30:16.7248721Z 2024-06-26T08:30:18.9611661Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:18.9612769Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:30:18.9613748Z cuda train mnasnet1_0 2024-06-26T08:30:26.3487595Z W0626 08:30:26.348000 140486004544128 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:30:43.7692635Z pass 2024-06-26T08:30:43.7734441Z TIMING: entire_frame_compile:22.87022 backend_compile:13.76616 2024-06-26T08:30:43.7735570Z STATS: call_* op count: 648 | FakeTensor.__torch_dispatch__:4228 | FakeTensorMode.__torch_dispatch__:47891 | ProxyTorchDispatchMode.__torch_dispatch__:12781 2024-06-26T08:30:43.7738882Z Dynamo produced 3 graphs covering 648 ops with 7 graph breaks (5 unique) 2024-06-26T08:30:47.8274424Z 2024-06-26T08:30:49.6134993Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:49.6135512Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:30:49.6136018Z cuda train mobilenet_v2 2024-06-26T08:30:57.1302598Z pass 2024-06-26T08:30:57.1348469Z TIMING: entire_frame_compile:5.71493 backend_compile:3.88428 2024-06-26T08:30:57.1350623Z STATS: call_* op count: 157 | FakeTensor.__torch_dispatch__:1035 | FakeTensorMode.__torch_dispatch__:23958 | ProxyTorchDispatchMode.__torch_dispatch__:5414 2024-06-26T08:30:57.1353310Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-06-26T08:31:00.6256207Z 2024-06-26T08:31:02.4093151Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:02.4093767Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:31:02.4094287Z cuda train mobilenet_v2_quantized_qat 2024-06-26T08:31:02.4094753Z Traceback (most recent call last): 2024-06-26T08:31:02.4095475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:31:02.4096230Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:31:02.4097155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:31:02.4098063Z pred = mod(*cloned_inputs) 2024-06-26T08:31:02.4099128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:31:02.4100117Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:31:02.4101186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:31:02.4102067Z raise e 2024-06-26T08:31:02.4102864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:31:02.4104183Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:31:02.4105559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:31:02.4106721Z return self._call_impl(*args, **kwargs) 2024-06-26T08:31:02.4107916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:02.4108746Z return forward_call(*args, **kwargs) 2024-06-26T08:31:02.4109212Z File ".3", line 207, in forward 2024-06-26T08:31:02.4109945Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-06-26T08:31:02.4111166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:31:02.4112051Z return self._call_impl(*args, **kwargs) 2024-06-26T08:31:02.4112963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:02.4113790Z return forward_call(*args, **kwargs) 2024-06-26T08:31:02.4114736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:31:02.4118092Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:31:02.4118904Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:31:02.4119285Z 2024-06-26T08:31:02.4119586Z The above exception was the direct cause of the following exception: 2024-06-26T08:31:02.4120027Z 2024-06-26T08:31:02.4120170Z Traceback (most recent call last): 2024-06-26T08:31:02.4120827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:31:02.4121481Z ) = runner.load_model( 2024-06-26T08:31:02.4122248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:31:02.4123001Z self.validate_model(model, example_inputs) 2024-06-26T08:31:02.4123753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:31:02.4124514Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:31:02.4124979Z RuntimeError: Eager run failed 2024-06-26T08:31:02.4125232Z 2024-06-26T08:31:02.4125353Z eager_fail_to_run 2024-06-26T08:31:05.6777194Z 2024-06-26T08:31:07.8762327Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:07.8763215Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:31:07.8764075Z cuda train mobilenet_v3_large 2024-06-26T08:31:16.0455514Z W0626 08:31:16.044000 139623805698688 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:31:35.6393446Z pass 2024-06-26T08:31:35.6442053Z TIMING: entire_frame_compile:25.66304 backend_compile:15.52516 2024-06-26T08:31:35.6447137Z STATS: call_* op count: 731 | FakeTensor.__torch_dispatch__:4708 | FakeTensorMode.__torch_dispatch__:51330 | ProxyTorchDispatchMode.__torch_dispatch__:14109 2024-06-26T08:31:35.6448338Z Dynamo produced 3 graphs covering 731 ops with 7 graph breaks (5 unique) 2024-06-26T08:31:39.5546437Z 2024-06-26T08:31:41.5141775Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:41.5142235Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:31:41.5142690Z cuda train moco 2024-06-26T08:31:46.0933742Z [rank0]:W0626 08:31:46.092000 139665901023872 torch/_logging/_internal.py:1040] [3/0] Profiler function will be ignored 2024-06-26T08:31:51.8891538Z [rank0]:W0626 08:31:51.888000 139665901023872 torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:31:51.9080555Z [rank0]:W0626 08:31:51.907000 139665901023872 torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:31:51.9087876Z [rank0]:W0626 08:31:51.907000 139665901023872 torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:31:51.9089420Z [rank0]:W0626 08:31:51.907000 139665901023872 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:31:51.9090952Z [rank0]:W0626 08:31:51.907000 139665901023872 torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:31:51.9092462Z [rank0]:W0626 08:31:51.907000 139665901023872 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ 0 │ 161 │ 94032128 │ 2024-06-26T08:31:51.9093947Z [rank0]:W0626 08:31:51.907000 139665901023872 torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:32:02.5658075Z [rank0]:W0626 08:32:02.565000 139665901023872 torch/_dynamo/variables/tensor.py:714] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:32:02.5660279Z [rank0]:W0626 08:32:02.565000 139665901023872 torch/_dynamo/variables/tensor.py:714] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:32:02.5661943Z [rank0]:W0626 08:32:02.565000 139665901023872 torch/_dynamo/variables/tensor.py:714] [7/0] or: 2024-06-26T08:32:02.5663720Z [rank0]:W0626 08:32:02.565000 139665901023872 torch/_dynamo/variables/tensor.py:714] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:32:02.5665495Z [rank0]:W0626 08:32:02.565000 139665901023872 torch/_dynamo/variables/tensor.py:714] [7/0] to include these operations in the captured graph. 2024-06-26T08:32:02.5666906Z [rank0]:W0626 08:32:02.565000 139665901023872 torch/_dynamo/variables/tensor.py:714] [7/0] 2024-06-26T08:32:06.7681969Z pass 2024-06-26T08:32:06.7979190Z TIMING: entire_frame_compile:20.05482 backend_compile:11.91289 2024-06-26T08:32:06.7980512Z STATS: call_* op count: 887 | FakeTensorMode.__torch_dispatch__:57161 | FakeTensor.__torch_dispatch__:2547 | ProxyTorchDispatchMode.__torch_dispatch__:13601 2024-06-26T08:32:06.7981658Z Dynamo produced 10 graphs covering 887 ops with 11 graph breaks (8 unique) 2024-06-26T08:32:07.3121125Z [rank0]:[W626 08:32:07.349080711 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:32:11.0460134Z 2024-06-26T08:32:12.9142353Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:32:13.1308720Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:32:13.1309609Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:32:13.1312031Z using fused AdamW: True 2024-06-26T08:32:13.4603633Z 2024-06-26T08:32:13.4604888Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:32:13.4605648Z cuda train nanogpt 2024-06-26T08:32:21.7956116Z W0626 08:32:21.794000 140687654912640 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:32:36.0773991Z pass 2024-06-26T08:32:36.1826542Z TIMING: entire_frame_compile:20.80675 backend_compile:13.15409 2024-06-26T08:32:36.1830120Z STATS: call_* op count: 859 | FakeTensorMode.__torch_dispatch__:45496 | FakeTensor.__torch_dispatch__:3618 | ProxyTorchDispatchMode.__torch_dispatch__:15889 2024-06-26T08:32:36.1831351Z Dynamo produced 3 graphs covering 859 ops with 7 graph breaks (5 unique) 2024-06-26T08:32:39.8699542Z 2024-06-26T08:32:42.1846103Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:42.1846571Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:32:42.1847025Z cuda train nvidia_deeprecommender 2024-06-26T08:32:43.5767925Z W0626 08:32:43.575000 139962018079360 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:32:45.0430477Z pass 2024-06-26T08:32:45.0729310Z TIMING: entire_frame_compile:1.91835 backend_compile:1.12273 2024-06-26T08:32:45.0730397Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:3256 | FakeTensor.__torch_dispatch__:283 | ProxyTorchDispatchMode.__torch_dispatch__:1070 2024-06-26T08:32:45.0731511Z Dynamo produced 3 graphs covering 71 ops with 7 graph breaks (5 unique) 2024-06-26T08:32:48.3795239Z 2024-06-26T08:32:49.1791211Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:49.1791695Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:49.1792138Z cuda train opacus_cifar10 2024-06-26T08:32:49.1816296Z Traceback (most recent call last): 2024-06-26T08:32:49.1817020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:32:49.1817802Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:32:49.1818652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:32:49.1819496Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:32:49.1820464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:32:49.1821216Z torch.autograd.backward( 2024-06-26T08:32:49.1822058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:32:49.1822831Z _engine_run_backward( 2024-06-26T08:32:49.1823950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:32:49.1825107Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:32:49.1826290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 97, in __call__ 2024-06-26T08:32:49.1827124Z return self.hook(module, *args, **kwargs) 2024-06-26T08:32:49.1828344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook 2024-06-26T08:32:49.1829800Z activations, backprops = self.rearrange_grad_samples( 2024-06-26T08:32:49.1831035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples 2024-06-26T08:32:49.1832055Z raise ValueError( 2024-06-26T08:32:49.1832963Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-06-26T08:32:49.1833644Z 2024-06-26T08:32:49.1833943Z The above exception was the direct cause of the following exception: 2024-06-26T08:32:49.1834383Z 2024-06-26T08:32:49.1834529Z Traceback (most recent call last): 2024-06-26T08:32:49.1835251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:32:49.1835916Z ) = runner.load_model( 2024-06-26T08:32:49.1836576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:32:49.1837324Z self.validate_model(model, example_inputs) 2024-06-26T08:32:49.1838083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:32:49.1838911Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:32:49.1839392Z RuntimeError: Eager run failed 2024-06-26T08:32:49.1839646Z 2024-06-26T08:32:49.1839768Z eager_fail_to_run 2024-06-26T08:32:52.4233647Z 2024-06-26T08:32:53.1036414Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:53.1036923Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:53.1037434Z cuda train phlippe_densenet 2024-06-26T08:33:00.3892779Z pass 2024-06-26T08:33:00.3895043Z TIMING: entire_frame_compile:6.6068 backend_compile:4.49547 2024-06-26T08:33:00.3896776Z STATS: call_* op count: 190 | FakeTensor.__torch_dispatch__:1014 | FakeTensorMode.__torch_dispatch__:26086 | ProxyTorchDispatchMode.__torch_dispatch__:6410 2024-06-26T08:33:00.3897931Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:03.8655470Z 2024-06-26T08:33:04.4549857Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:04.4550360Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:04.4550815Z cuda train phlippe_resnet 2024-06-26T08:33:07.3336959Z pass 2024-06-26T08:33:07.3337682Z TIMING: entire_frame_compile:2.49526 backend_compile:1.72659 2024-06-26T08:33:07.3339315Z STATS: call_* op count: 75 | FakeTensor.__torch_dispatch__:412 | FakeTensorMode.__torch_dispatch__:9786 | ProxyTorchDispatchMode.__torch_dispatch__:2419 2024-06-26T08:33:07.3341039Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:10.7156623Z 2024-06-26T08:33:11.8290380Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:11.8290889Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:33:11.8292416Z cuda train pytorch_CycleGAN_and_pix2pix 2024-06-26T08:33:23.3734004Z pass 2024-06-26T08:33:23.3822014Z TIMING: entire_frame_compile:2.66896 backend_compile:1.97313 2024-06-26T08:33:23.3823114Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:9722 | FakeTensor.__torch_dispatch__:321 | ProxyTorchDispatchMode.__torch_dispatch__:3021 2024-06-26T08:33:23.3824458Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:26.7702579Z 2024-06-26T08:33:28.1110830Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:28.1111305Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:33:28.1111762Z cuda train pytorch_stargan 2024-06-26T08:33:57.4137844Z pass 2024-06-26T08:33:57.4322390Z TIMING: entire_frame_compile:3.09901 backend_compile:2.36565 2024-06-26T08:33:57.4325520Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:12623 | FakeTensor.__torch_dispatch__:356 | ProxyTorchDispatchMode.__torch_dispatch__:3815 2024-06-26T08:33:57.4326698Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-06-26T08:34:00.8508646Z 2024-06-26T08:34:01.6312195Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:01.6312863Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:01.6313466Z cuda train pytorch_unet 2024-06-26T08:34:24.6488400Z W0626 08:34:24.647000 139972201820800 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:34:32.7061342Z pass_due_to_skip 2024-06-26T08:34:32.8798400Z TIMING: entire_frame_compile:10.98154 backend_compile:6.76288 2024-06-26T08:34:32.8799560Z STATS: call_* op count: 315 | FakeTensor.__torch_dispatch__:1963 | FakeTensorMode.__torch_dispatch__:23791 | ProxyTorchDispatchMode.__torch_dispatch__:7044 2024-06-26T08:34:32.8800667Z Dynamo produced 3 graphs covering 315 ops with 7 graph breaks (5 unique) 2024-06-26T08:34:36.3617422Z 2024-06-26T08:34:38.5061915Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:38.5062374Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:34:38.5062809Z cuda train resnet152 2024-06-26T08:35:00.6321183Z W0626 08:35:00.631000 140087184663168 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:35:51.9322166Z pass 2024-06-26T08:35:51.9968895Z TIMING: entire_frame_compile:66.77503 backend_compile:41.05425 2024-06-26T08:35:51.9969998Z STATS: call_* op count: 1938 | FakeTensor.__torch_dispatch__:11838 | FakeTensorMode.__torch_dispatch__:142379 | ProxyTorchDispatchMode.__torch_dispatch__:38630 2024-06-26T08:35:51.9974066Z Dynamo produced 3 graphs covering 1938 ops with 7 graph breaks (5 unique) 2024-06-26T08:35:56.8723158Z 2024-06-26T08:35:57.9265436Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:57.9266005Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:35:57.9266935Z cuda train resnet18 2024-06-26T08:36:02.0062815Z pass 2024-06-26T08:36:02.0141032Z TIMING: entire_frame_compile:2.51163 backend_compile:1.75089 2024-06-26T08:36:02.0142678Z STATS: call_* op count: 73 | FakeTensor.__torch_dispatch__:410 | FakeTensorMode.__torch_dispatch__:9807 | ProxyTorchDispatchMode.__torch_dispatch__:2338 2024-06-26T08:36:02.0144725Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-06-26T08:36:05.3975108Z 2024-06-26T08:36:06.9985579Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:06.9986134Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:36:06.9986569Z cuda train resnet50 2024-06-26T08:36:17.2664236Z pass 2024-06-26T08:36:17.2848066Z TIMING: entire_frame_compile:5.92613 backend_compile:4.28755 2024-06-26T08:36:17.2850351Z STATS: call_* op count: 179 | FakeTensor.__torch_dispatch__:1015 | FakeTensorMode.__torch_dispatch__:25360 | ProxyTorchDispatchMode.__torch_dispatch__:6038 2024-06-26T08:36:17.2851529Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-06-26T08:36:20.7296973Z 2024-06-26T08:36:22.7540131Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:22.7540686Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:36:22.7542300Z cuda train resnet50_quantized_qat 2024-06-26T08:36:22.7542882Z Traceback (most recent call last): 2024-06-26T08:36:22.7544108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:36:22.7545056Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:36:22.7546189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:36:22.7547221Z pred = mod(*cloned_inputs) 2024-06-26T08:36:22.7548392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:36:22.7549278Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:36:22.7550214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:36:22.7551049Z raise e 2024-06-26T08:36:22.7552085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:36:22.7553353Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:36:22.7554895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:36:22.7555798Z return self._call_impl(*args, **kwargs) 2024-06-26T08:36:22.7556731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:36:22.7557558Z return forward_call(*args, **kwargs) 2024-06-26T08:36:22.7558029Z File ".3", line 167, in forward 2024-06-26T08:36:22.7558664Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-06-26T08:36:22.7559804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:36:22.7560682Z return self._call_impl(*args, **kwargs) 2024-06-26T08:36:22.7561715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:36:22.7562609Z return forward_call(*args, **kwargs) 2024-06-26T08:36:22.7563571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:36:22.7564474Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:36:22.7565014Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:36:22.7565393Z 2024-06-26T08:36:22.7565688Z The above exception was the direct cause of the following exception: 2024-06-26T08:36:22.7566132Z 2024-06-26T08:36:22.7566284Z Traceback (most recent call last): 2024-06-26T08:36:22.7566926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:36:22.7567695Z ) = runner.load_model( 2024-06-26T08:36:22.7568349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:36:22.7569104Z self.validate_model(model, example_inputs) 2024-06-26T08:36:22.7569858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:36:22.7570610Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:36:22.7571076Z RuntimeError: Eager run failed 2024-06-26T08:36:22.7571336Z 2024-06-26T08:36:22.7571454Z eager_fail_to_run 2024-06-26T08:36:26.0262766Z 2024-06-26T08:36:27.7582837Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:27.7583533Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:36:27.7583985Z cuda train resnext50_32x4d 2024-06-26T08:36:39.4939914Z W0626 08:36:39.493000 140498355176064 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:36:57.3969862Z pass 2024-06-26T08:36:57.4169319Z TIMING: entire_frame_compile:23.48883 backend_compile:14.33836 2024-06-26T08:36:57.4170488Z STATS: call_* op count: 680 | FakeTensor.__torch_dispatch__:4178 | FakeTensorMode.__torch_dispatch__:49216 | ProxyTorchDispatchMode.__torch_dispatch__:13300 2024-06-26T08:36:57.4172603Z Dynamo produced 3 graphs covering 680 ops with 7 graph breaks (5 unique) 2024-06-26T08:37:01.2498113Z 2024-06-26T08:37:07.1254412Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:07.1255215Z loading model: 0it [00:05, ?it/s] 2024-06-26T08:37:07.1255695Z cuda train sam 2024-06-26T08:37:07.1261428Z Traceback (most recent call last): 2024-06-26T08:37:07.1262255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:37:07.1263087Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:37:07.1264179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:37:07.1265118Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:37:07.1266210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:37:07.1266997Z torch.autograd.backward( 2024-06-26T08:37:07.1268241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:37:07.1269213Z _engine_run_backward( 2024-06-26T08:37:07.1270234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:37:07.1271375Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:37:07.1272319Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-06-26T08:37:07.1272833Z 2024-06-26T08:37:07.1273127Z The above exception was the direct cause of the following exception: 2024-06-26T08:37:07.1273574Z 2024-06-26T08:37:07.1273733Z Traceback (most recent call last): 2024-06-26T08:37:07.1274382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:37:07.1275032Z ) = runner.load_model( 2024-06-26T08:37:07.1275694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:37:07.1276444Z self.validate_model(model, example_inputs) 2024-06-26T08:37:07.1277193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:37:07.1277946Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:37:07.1278412Z RuntimeError: Eager run failed 2024-06-26T08:37:07.1278666Z 2024-06-26T08:37:07.1278790Z eager_fail_to_run 2024-06-26T08:37:10.3932529Z 2024-06-26T08:37:11.4676625Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:11.4677206Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:37:11.4678039Z cuda train shufflenet_v2_x1_0 2024-06-26T08:37:19.0669092Z pass 2024-06-26T08:37:19.0678456Z TIMING: entire_frame_compile:6.61475 backend_compile:4.68896 2024-06-26T08:37:19.0679715Z STATS: call_* op count: 275 | FakeTensor.__torch_dispatch__:1058 | FakeTensorMode.__torch_dispatch__:28325 | ProxyTorchDispatchMode.__torch_dispatch__:6897 2024-06-26T08:37:19.0680921Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique) 2024-06-26T08:37:22.5398592Z 2024-06-26T08:37:23.2375063Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:23.2375535Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:23.2375979Z cuda train soft_actor_critic 2024-06-26T08:37:24.0345751Z W0626 08:37:24.033000 139725948670592 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:37:24.6860155Z pass 2024-06-26T08:37:24.6860833Z TIMING: entire_frame_compile:1.22947 backend_compile:0.69987 2024-06-26T08:37:24.6862422Z STATS: call_* op count: 57 | FakeTensor.__torch_dispatch__:198 | FakeTensorMode.__torch_dispatch__:2192 | ProxyTorchDispatchMode.__torch_dispatch__:686 2024-06-26T08:37:24.6864287Z Dynamo produced 4 graphs covering 57 ops with 6 graph breaks (5 unique) 2024-06-26T08:37:28.0061311Z 2024-06-26T08:37:29.1905037Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:29.1905748Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:37:29.1906464Z cuda train speech_transformer 2024-06-26T08:37:43.0076015Z pass 2024-06-26T08:37:43.0501717Z TIMING: entire_frame_compile:11.13741 backend_compile:8.37187 2024-06-26T08:37:43.0504195Z STATS: call_* op count: 771 | FakeTensorMode.__torch_dispatch__:39144 | FakeTensor.__torch_dispatch__:859 | ProxyTorchDispatchMode.__torch_dispatch__:17627 2024-06-26T08:37:43.0505975Z Dynamo produced 10 graphs covering 771 ops with 16 graph breaks (7 unique) 2024-06-26T08:37:46.6455536Z 2024-06-26T08:37:47.6036505Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:47.6037140Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:47.6037581Z cuda train squeezenet1_1 2024-06-26T08:37:50.5260013Z pass 2024-06-26T08:37:50.5273954Z TIMING: entire_frame_compile:1.81132 backend_compile:1.27757 2024-06-26T08:37:50.5275426Z STATS: call_* op count: 70 | FakeTensor.__torch_dispatch__:359 | FakeTensorMode.__torch_dispatch__:5509 | ProxyTorchDispatchMode.__torch_dispatch__:1964 2024-06-26T08:37:50.5276533Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-06-26T08:37:53.9043163Z 2024-06-26T08:37:55.0891324Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:55.0891720Z 2024-06-26T08:37:55.3847558Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-06-26T08:39:05.0643892Z pass 2024-06-26T08:39:05.0694976Z TIMING: entire_frame_compile:34.11684 backend_compile:19.3872 2024-06-26T08:39:05.0696232Z STATS: call_* op count: 974 | FakeTensor.__torch_dispatch__:5135 | FakeTensorMode.__torch_dispatch__:61524 | ProxyTorchDispatchMode.__torch_dispatch__:18598 2024-06-26T08:39:05.0697334Z Dynamo produced 3 graphs covering 974 ops with 7 graph breaks (5 unique) 2024-06-26T08:39:09.3078476Z 2024-06-26T08:39:11.8210438Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:39:11.8211488Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:39:34.8348563Z cuda train timm_regnet 2024-06-26T08:39:34.8349032Z pass 2024-06-26T08:39:34.8763756Z TIMING: entire_frame_compile:13.00605 backend_compile:6.79308 2024-06-26T08:39:34.8764866Z STATS: call_* op count: 463 | FakeTensor.__torch_dispatch__:897 | FakeTensorMode.__torch_dispatch__:36256 | ProxyTorchDispatchMode.__torch_dispatch__:10183 2024-06-26T08:39:34.8767279Z Dynamo produced 2 graphs covering 463 ops with 6 graph breaks (5 unique) 2024-06-26T08:39:38.5480553Z 2024-06-26T08:39:40.2947495Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:39:40.2948439Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:39:40.2948895Z cuda train timm_resnest 2024-06-26T08:39:48.0044647Z W0626 08:39:48.003000 140225699312256 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:39:57.3494083Z pass 2024-06-26T08:39:57.3566331Z TIMING: entire_frame_compile:12.83952 backend_compile:7.82423 2024-06-26T08:39:57.3567490Z STATS: call_* op count: 419 | FakeTensor.__torch_dispatch__:2248 | FakeTensorMode.__torch_dispatch__:26147 | ProxyTorchDispatchMode.__torch_dispatch__:7360 2024-06-26T08:39:57.3568627Z Dynamo produced 3 graphs covering 419 ops with 7 graph breaks (5 unique) 2024-06-26T08:40:01.1364347Z 2024-06-26T08:40:02.5421187Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:02.5422328Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:02.5425600Z cuda train timm_vision_transformer 2024-06-26T08:40:09.4399998Z pass 2024-06-26T08:40:09.4549490Z TIMING: entire_frame_compile:5.62674 backend_compile:3.91629 2024-06-26T08:40:09.4550577Z STATS: call_* op count: 355 | FakeTensor.__torch_dispatch__:1075 | FakeTensorMode.__torch_dispatch__:20017 | ProxyTorchDispatchMode.__torch_dispatch__:7504 2024-06-26T08:40:09.4551686Z Dynamo produced 2 graphs covering 355 ops with 6 graph breaks (5 unique) 2024-06-26T08:40:13.0214794Z 2024-06-26T08:40:25.3982817Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:25.3983565Z loading model: 0it [00:12, ?it/s] 2024-06-26T08:40:25.3984136Z cuda train timm_vision_transformer_large 2024-06-26T08:40:25.3986635Z pass_due_to_skip 2024-06-26T08:40:25.6615361Z TIMING: 2024-06-26T08:40:25.6615769Z STATS: call_* op count: 0 2024-06-26T08:40:25.6616319Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:40:29.0454193Z 2024-06-26T08:40:30.8404386Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:30.8404901Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:30.8407147Z cuda train timm_vovnet 2024-06-26T08:40:44.0863110Z pass 2024-06-26T08:40:44.1031952Z TIMING: entire_frame_compile:7.05391 backend_compile:3.65911 2024-06-26T08:40:44.1033025Z STATS: call_* op count: 174 | FakeTensor.__torch_dispatch__:287 | FakeTensorMode.__torch_dispatch__:18221 | ProxyTorchDispatchMode.__torch_dispatch__:4649 2024-06-26T08:40:44.1034127Z Dynamo produced 2 graphs covering 174 ops with 6 graph breaks (5 unique) 2024-06-26T08:40:47.6935412Z 2024-06-26T08:40:49.4262398Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:49.4262961Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:49.4263745Z cuda train torch_multimodal_clip 2024-06-26T08:41:05.9294476Z W0626 08:41:05.928000 139898045084288 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:41:35.0320399Z pass 2024-06-26T08:41:35.1619960Z TIMING: entire_frame_compile:40.44127 backend_compile:27.26225 2024-06-26T08:41:35.1621064Z STATS: call_* op count: 961 | FakeTensor.__torch_dispatch__:7961 | FakeTensorMode.__torch_dispatch__:93900 | ProxyTorchDispatchMode.__torch_dispatch__:32713 2024-06-26T08:41:35.1622177Z Dynamo produced 3 graphs covering 961 ops with 7 graph breaks (5 unique) 2024-06-26T08:41:39.3969249Z 2024-06-26T08:41:39.9353948Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:39.9354428Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:39.9355789Z cuda train tts_angular 2024-06-26T08:41:40.7335942Z W0626 08:41:40.732000 140301461922432 torch/_logging/_internal.py:1040] [10/0] Profiler function will be ignored 2024-06-26T08:41:42.2805512Z pass 2024-06-26T08:41:42.2907311Z TIMING: entire_frame_compile:1.89145 backend_compile:1.10687 2024-06-26T08:41:42.2908396Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:2622 | FakeTensor.__torch_dispatch__:348 | ProxyTorchDispatchMode.__torch_dispatch__:825 2024-06-26T08:41:42.2909801Z Dynamo produced 5 graphs covering 71 ops with 9 graph breaks (6 unique) 2024-06-26T08:41:45.6053473Z 2024-06-26T08:41:47.7521758Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:47.7522389Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:41:47.7522869Z cuda train vgg16 2024-06-26T08:42:06.1980089Z pass 2024-06-26T08:42:06.2232333Z TIMING: entire_frame_compile:1.08706 backend_compile:0.74485 2024-06-26T08:42:06.2233477Z STATS: call_* op count: 44 | FakeTensor.__torch_dispatch__:230 | FakeTensorMode.__torch_dispatch__:3546 | ProxyTorchDispatchMode.__torch_dispatch__:1238 2024-06-26T08:42:06.2234634Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-06-26T08:42:09.5799170Z 2024-06-26T08:42:12.6556196Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:12.6556663Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:42:12.6557100Z cuda train vision_maskrcnn 2024-06-26T08:42:37.4077979Z W0626 08:42:37.407000 140437114733184 torch/_dynamo/variables/tensor.py:714] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:42:37.4080167Z W0626 08:42:37.407000 140437114733184 torch/_dynamo/variables/tensor.py:714] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:42:37.4082537Z W0626 08:42:37.407000 140437114733184 torch/_dynamo/variables/tensor.py:714] [18/0] or: 2024-06-26T08:42:37.4083837Z W0626 08:42:37.407000 140437114733184 torch/_dynamo/variables/tensor.py:714] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:42:37.4085108Z W0626 08:42:37.407000 140437114733184 torch/_dynamo/variables/tensor.py:714] [18/0] to include these operations in the captured graph. 2024-06-26T08:42:37.4086479Z W0626 08:42:37.407000 140437114733184 torch/_dynamo/variables/tensor.py:714] [18/0] 2024-06-26T08:43:34.2011335Z W0626 08:43:34.200000 140437114733184 torch/_dynamo/convert_frame.py:762] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-06-26T08:43:34.2014147Z W0626 08:43:34.200000 140437114733184 torch/_dynamo/convert_frame.py:762] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-06-26T08:43:34.2016799Z W0626 08:43:34.200000 140437114733184 torch/_dynamo/convert_frame.py:762] [30/8] last reason: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 619 2024-06-26T08:43:34.2019012Z W0626 08:43:34.200000 140437114733184 torch/_dynamo/convert_frame.py:762] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-06-26T08:43:34.2021443Z W0626 08:43:34.200000 140437114733184 torch/_dynamo/convert_frame.py:762] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-06-26T08:44:05.3246191Z pass 2024-06-26T08:44:05.3908448Z TIMING: entire_frame_compile:104.12883 backend_compile:62.96277 2024-06-26T08:44:05.3910862Z STATS: call_* op count: 2515 | FakeTensorMode.__torch_dispatch__:129988 | FakeTensor.__torch_dispatch__:643 | ProxyTorchDispatchMode.__torch_dispatch__:41391 | attempt fast:1873 | slow no contiguity match:182 | fast is_contiguous:1331 | slow both tensors nontrivially broadcast:360 2024-06-26T08:44:05.3913336Z Dynamo produced 51 graphs covering 2515 ops with 33 graph breaks (8 unique) 2024-06-26T08:44:12.1011110Z 2024-06-26T08:44:14.7328351Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:14.7328971Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:44:14.7330655Z cuda train yolov3 2024-06-26T08:44:43.3037441Z W0626 08:44:43.302000 140035466756736 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:45:17.1541589Z E0626 08:45:17.153000 140035466756736 torch/_dynamo/utils.py:1470] RMSE (res-fp64): 0.21506, (ref-fp64): 0.07074 and shape=torch.Size([32, 3, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-26T08:45:17.1543782Z E0626 08:45:17.153000 140035466756736 torch/_dynamo/utils.py:1384] Accuracy failed for key name module_list.0.Conv2d.weight.grad 2024-06-26T08:45:17.1573615Z fail_accuracy 2024-06-26T08:45:17.2176350Z TIMING: entire_frame_compile:43.89448 backend_compile:27.13503 2024-06-26T08:45:17.2177444Z STATS: call_* op count: 1232 | FakeTensor.__torch_dispatch__:7680 | FakeTensorMode.__torch_dispatch__:110094 | ProxyTorchDispatchMode.__torch_dispatch__:28065 2024-06-26T08:45:17.2178567Z Dynamo produced 4 graphs covering 1232 ops with 8 graph breaks (5 unique) 2024-06-26T08:45:19.0724742Z accuracy pass_rate=76.32% 2024-06-26T08:45:19.0728363Z calls_captured gmean=0.00x mean=427.500x 2024-06-26T08:45:19.0732504Z unique_graphs gmean=0.00x mean=3.816x 2024-06-26T08:45:19.0736389Z graph_breaks gmean=0.00x mean=6.395x 2024-06-26T08:45:19.0740381Z unique_graph_breaks gmean=0.00x mean=4.316x 2024-06-26T08:45:19.0744628Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:45:19.0748544Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:45:19.0752505Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T08:45:19.6282695Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_eager_torchbench_training.csv 2024-06-26T08:45:19.9563931Z llava XFAIL 2024-06-26T08:45:19.9567853Z maml_omniglot PASS 2024-06-26T08:45:19.9574197Z mnasnet1_0 PASS 2024-06-26T08:45:19.9579175Z mobilenet_v2 PASS 2024-06-26T08:45:19.9585290Z mobilenet_v2_quantized_qat XFAIL 2024-06-26T08:45:19.9590650Z mobilenet_v3_large PASS 2024-06-26T08:45:19.9596535Z moco PASS 2024-06-26T08:45:19.9602155Z nanogpt PASS 2024-06-26T08:45:19.9607977Z nvidia_deeprecommender PASS 2024-06-26T08:45:19.9613429Z opacus_cifar10 XFAIL 2024-06-26T08:45:19.9619354Z phlippe_densenet PASS 2024-06-26T08:45:19.9624968Z phlippe_resnet PASS 2024-06-26T08:45:19.9630614Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:45:19.9636136Z pytorch_stargan PASS 2024-06-26T08:45:19.9641556Z pytorch_unet XFAIL 2024-06-26T08:45:19.9647242Z resnet152 PASS 2024-06-26T08:45:19.9652622Z resnet18 PASS 2024-06-26T08:45:19.9658206Z resnet50 PASS 2024-06-26T08:45:19.9663962Z resnet50_quantized_qat XFAIL 2024-06-26T08:45:19.9669439Z resnext50_32x4d PASS 2024-06-26T08:45:19.9674944Z sam XFAIL 2024-06-26T08:45:19.9680222Z shufflenet_v2_x1_0 PASS 2024-06-26T08:45:19.9686047Z soft_actor_critic PASS 2024-06-26T08:45:19.9691432Z speech_transformer PASS 2024-06-26T08:45:19.9696867Z squeezenet1_1 PASS 2024-06-26T08:45:19.9702258Z stable_diffusion_text_encoder PASS 2024-06-26T08:45:19.9707951Z stable_diffusion_unet XFAIL 2024-06-26T08:45:19.9713345Z timm_efficientnet PASS 2024-06-26T08:45:19.9718715Z timm_regnet PASS 2024-06-26T08:45:19.9724486Z timm_resnest PASS 2024-06-26T08:45:19.9729769Z timm_vision_transformer PASS 2024-06-26T08:45:19.9735283Z timm_vision_transformer_large XFAIL 2024-06-26T08:45:19.9740656Z timm_vovnet PASS 2024-06-26T08:45:19.9746346Z torch_multimodal_clip PASS 2024-06-26T08:45:19.9751651Z tts_angular PASS 2024-06-26T08:45:19.9757025Z vgg16 PASS 2024-06-26T08:45:19.9762609Z vision_maskrcnn PASS 2024-06-26T08:45:19.9768126Z yolov3 XFAIL 2024-06-26T08:45:20.0223895Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_eager_torchbench_training.csv 2024-06-26T08:45:20.3588604Z llava PASS 2024-06-26T08:45:20.3591825Z maml_omniglot PASS 2024-06-26T08:45:20.3594932Z mnasnet1_0 PASS 2024-06-26T08:45:20.3599474Z mobilenet_v2 PASS 2024-06-26T08:45:20.3604168Z mobilenet_v2_quantized_qat PASS 2024-06-26T08:45:20.3608570Z mobilenet_v3_large PASS 2024-06-26T08:45:20.3613075Z moco PASS 2024-06-26T08:45:20.3617631Z nanogpt PASS 2024-06-26T08:45:20.3622148Z nvidia_deeprecommender PASS 2024-06-26T08:45:20.3626735Z opacus_cifar10 PASS 2024-06-26T08:45:20.3631213Z phlippe_densenet PASS 2024-06-26T08:45:20.3635685Z phlippe_resnet PASS 2024-06-26T08:45:20.3640097Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:45:20.3644827Z pytorch_stargan PASS 2024-06-26T08:45:20.3649198Z pytorch_unet PASS 2024-06-26T08:45:20.3653634Z resnet152 PASS 2024-06-26T08:45:20.3658109Z resnet18 PASS 2024-06-26T08:45:20.3662695Z resnet50 PASS 2024-06-26T08:45:20.3667336Z resnet50_quantized_qat PASS 2024-06-26T08:45:20.3671795Z resnext50_32x4d PASS 2024-06-26T08:45:20.3676150Z sam PASS 2024-06-26T08:45:20.3680678Z shufflenet_v2_x1_0 PASS 2024-06-26T08:45:20.3685332Z soft_actor_critic PASS 2024-06-26T08:45:20.3689651Z speech_transformer PASS 2024-06-26T08:45:20.3694084Z squeezenet1_1 PASS 2024-06-26T08:45:20.3698552Z stable_diffusion_text_encoder PASS 2024-06-26T08:45:20.3703009Z stable_diffusion_unet PASS 2024-06-26T08:45:20.3707802Z timm_efficientnet PASS 2024-06-26T08:45:20.3712196Z timm_regnet PASS 2024-06-26T08:45:20.3716669Z timm_resnest PASS 2024-06-26T08:45:20.3721052Z timm_vision_transformer PASS 2024-06-26T08:45:20.3725933Z timm_vision_transformer_large PASS 2024-06-26T08:45:20.3730207Z timm_vovnet PASS 2024-06-26T08:45:20.3734654Z torch_multimodal_clip PASS 2024-06-26T08:45:20.3738983Z tts_angular PASS 2024-06-26T08:45:20.3743732Z vgg16 PASS 2024-06-26T08:45:20.3748175Z vision_maskrcnn PASS 2024-06-26T08:45:20.3752576Z yolov3 PASS 2024-06-26T08:45:20.4209618Z + cleanup_workspace 2024-06-26T08:45:20.4210647Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:45:20.4212003Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:45:20.4212934Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:45:20.4213564Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:45:20.4214358Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:45:20.4215207Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:45:20.4215885Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:45:20.8838267Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:45:20.8838777Z cat test/**/*_toprint.log || true 2024-06-26T08:45:20.8846981Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:20.8847477Z env: 2024-06-26T08:45:20.8847753Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:20.8848299Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:20.8849042Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:20.8849683Z ##[endgroup] 2024-06-26T08:45:20.8913727Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:45:20.8940206Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:45:20.8940677Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:45:20.8947840Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:20.8948331Z env: 2024-06-26T08:45:20.8948604Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:20.8949054Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:20.8949785Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:20.8950445Z MONITOR_SCRIPT_PID: 13256 2024-06-26T08:45:20.8950795Z ##[endgroup] 2024-06-26T08:45:20.9134786Z Prepare all required actions 2024-06-26T08:45:20.9135228Z Getting action download info 2024-06-26T08:45:21.0770181Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:45:21.2427462Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:45:21.2427922Z with: 2024-06-26T08:45:21.2428451Z file-suffix: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330 2024-06-26T08:45:21.2429112Z s3-bucket: gha-artifacts 2024-06-26T08:45:21.2429449Z env: 2024-06-26T08:45:21.2429715Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:21.2430162Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:21.2430888Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:21.2431528Z ##[endgroup] 2024-06-26T08:45:21.2455837Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:45:21.2456586Z # Remove any previous test jsons if they exist 2024-06-26T08:45:21.2457096Z rm -f test-jsons-*.zip 2024-06-26T08:45:21.2457609Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:45:21.2465694Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:21.2466234Z env: 2024-06-26T08:45:21.2466514Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:21.2466967Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:21.2467737Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:21.2468629Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330 2024-06-26T08:45:21.2469270Z ##[endgroup] 2024-06-26T08:45:21.2639872Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:45:21.2670188Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:45:21.2671482Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:45:21.2675829Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:45:21.2701199Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:45:21.2701881Z # Remove any previous test reports if they exist 2024-06-26T08:45:21.2702394Z rm -f test-reports-*.zip 2024-06-26T08:45:21.2702974Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:45:21.2711017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:21.2711502Z env: 2024-06-26T08:45:21.2711783Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:21.2712231Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:21.2712964Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:21.2713848Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330 2024-06-26T08:45:21.2714490Z ##[endgroup] 2024-06-26T08:45:21.2872601Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-06-26T08:45:21.2874502Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2024-06-26T08:45:21.2875454Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 74%) 2024-06-26T08:45:21.2876322Z adding: test/test-reports/training_torchbench.csv (deflated 66%) 2024-06-26T08:45:21.2883023Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-26T08:45:21.2884690Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-26T08:45:21.2909644Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:45:21.2910265Z # Remove any previous usage logs if they exist 2024-06-26T08:45:21.2910769Z rm -f logs-*.zip 2024-06-26T08:45:21.2911431Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:45:21.2912406Z # so check to see if the file exists first 2024-06-26T08:45:21.2912914Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:45:21.2913444Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:45:21.2913928Z fi 2024-06-26T08:45:21.2914289Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:45:21.2914850Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:45:21.2915336Z fi 2024-06-26T08:45:21.2922878Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:21.2923371Z env: 2024-06-26T08:45:21.2923652Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:21.2924097Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:21.2924828Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:21.2925713Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330 2024-06-26T08:45:21.2926460Z ##[endgroup] 2024-06-26T08:45:21.3005551Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:45:21.3073775Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:45:21.3074493Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:45:21.3075049Z rm -f debug-*.zip 2024-06-26T08:45:21.3075433Z if [ -d 'test/debug' ]; then 2024-06-26T08:45:21.3075937Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:45:21.3076420Z fi 2024-06-26T08:45:21.3083704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:21.3084208Z env: 2024-06-26T08:45:21.3084483Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:21.3084942Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:21.3085681Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:21.3086581Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330 2024-06-26T08:45:21.3087234Z ##[endgroup] 2024-06-26T08:45:21.3164622Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:45:21.3165063Z with: 2024-06-26T08:45:21.3165349Z s3-bucket: gha-artifacts 2024-06-26T08:45:21.3165769Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:45:21.3166238Z retention-days: 14 2024-06-26T08:45:21.3166565Z if-no-files-found: warn 2024-06-26T08:45:21.3166924Z path: test-jsons-*.zip 2024-06-26T08:45:21.3167273Z name: artifact 2024-06-26T08:45:21.3167573Z region: us-east-1 2024-06-26T08:45:21.3167880Z env: 2024-06-26T08:45:21.3168168Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:21.3168626Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:21.3169351Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:21.3169990Z ##[endgroup] 2024-06-26T08:45:21.6254402Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:45:21.6255248Z With the provided path, there will be 1 file uploaded 2024-06-26T08:45:21.6255886Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:45:21.6281661Z Starting upload of test-jsons-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330.zip 2024-06-26T08:45:21.7595832Z Finished upload of test-jsons-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330.zip 2024-06-26T08:45:21.7735808Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:45:21.7736242Z with: 2024-06-26T08:45:21.7736528Z s3-bucket: gha-artifacts 2024-06-26T08:45:21.7736954Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:45:21.7737424Z retention-days: 14 2024-06-26T08:45:21.7737756Z if-no-files-found: error 2024-06-26T08:45:21.7738122Z path: test-reports-*.zip 2024-06-26T08:45:21.7738471Z name: artifact 2024-06-26T08:45:21.7738768Z region: us-east-1 2024-06-26T08:45:21.7739070Z env: 2024-06-26T08:45:21.7739344Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:21.7739811Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:21.7740661Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:21.7741305Z ##[endgroup] 2024-06-26T08:45:22.0845310Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:45:22.0846601Z With the provided path, there will be 1 file uploaded 2024-06-26T08:45:22.0847831Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:45:22.0871581Z Starting upload of test-reports-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330.zip 2024-06-26T08:45:22.1997711Z Finished upload of test-reports-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330.zip 2024-06-26T08:45:22.2141071Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:45:22.2141517Z with: 2024-06-26T08:45:22.2141793Z s3-bucket: gha-artifacts 2024-06-26T08:45:22.2142249Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:45:22.2142868Z retention-days: 14 2024-06-26T08:45:22.2143203Z if-no-files-found: ignore 2024-06-26T08:45:22.2143861Z path: logs-*.zip 2024-06-26T08:45:22.2144173Z name: artifact 2024-06-26T08:45:22.2144473Z region: us-east-1 2024-06-26T08:45:22.2144761Z env: 2024-06-26T08:45:22.2145032Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:22.2145477Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:22.2146202Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:22.2146831Z ##[endgroup] 2024-06-26T08:45:22.5291961Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:45:22.5292858Z With the provided path, there will be 1 file uploaded 2024-06-26T08:45:22.5293673Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:45:22.5320436Z Starting upload of logs-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330.zip 2024-06-26T08:45:22.6891237Z Finished upload of logs-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688916330.zip 2024-06-26T08:45:22.7029004Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:45:22.7029438Z with: 2024-06-26T08:45:22.7029731Z s3-bucket: gha-artifacts 2024-06-26T08:45:22.7030160Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:45:22.7030637Z retention-days: 14 2024-06-26T08:45:22.7030976Z if-no-files-found: ignore 2024-06-26T08:45:22.7031337Z path: debug-*.zip 2024-06-26T08:45:22.7031654Z name: artifact 2024-06-26T08:45:22.7031952Z region: us-east-1 2024-06-26T08:45:22.7032256Z env: 2024-06-26T08:45:22.7032545Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:22.7033056Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:22.7033787Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:22.7034427Z ##[endgroup] 2024-06-26T08:45:23.0057911Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:45:23.0197211Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:45:23.0197694Z # shellcheck disable=SC2156 2024-06-26T08:45:23.0198474Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:45:23.0206694Z shell: /usr/bin/bash -e {0} 2024-06-26T08:45:23.0207043Z env: 2024-06-26T08:45:23.0207317Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:23.0207763Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:23.0208490Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:23.0209127Z ##[endgroup] 2024-06-26T08:45:23.1860998Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:45:23.1861562Z with: 2024-06-26T08:45:23.1861818Z env: 2024-06-26T08:45:23.1862096Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:23.1862591Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:23.1863509Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:23.1864206Z ##[endgroup] 2024-06-26T08:45:23.1881892Z ##[group]Run set -eou pipefail 2024-06-26T08:45:23.1882298Z set -eou pipefail 2024-06-26T08:45:23.1882650Z  2024-06-26T08:45:23.1883148Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:45:23.1883769Z for _ in $(seq 1440); do 2024-06-26T08:45:23.1884240Z  # Break if no ssh session exists anymore 2024-06-26T08:45:23.1884727Z  if [ "$(who)" = "" ]; then 2024-06-26T08:45:23.1885128Z  break 2024-06-26T08:45:23.1885462Z  fi 2024-06-26T08:45:23.1885753Z  echo "." 2024-06-26T08:45:23.1886080Z  sleep 5 2024-06-26T08:45:23.1886393Z done 2024-06-26T08:45:23.1893714Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:23.1894203Z env: 2024-06-26T08:45:23.1894486Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:23.1895062Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:23.1895788Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:23.1896415Z ##[endgroup] 2024-06-26T08:45:23.1915961Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:45:23.1953803Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:45:23.1954553Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:45:23.1955134Z # shellcheck disable=SC2046 2024-06-26T08:45:23.1955600Z docker stop $(docker ps -q) || true 2024-06-26T08:45:23.1956067Z # Prune all of the docker images 2024-06-26T08:45:23.1956513Z docker system prune -af 2024-06-26T08:45:23.1963532Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:23.1964028Z env: 2024-06-26T08:45:23.1964299Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:45:23.1964758Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:45:23.1965490Z DOCKER_CONTAINER_ID: 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:23.1966127Z ##[endgroup] 2024-06-26T08:45:23.5593136Z 255a27a1a4ef 2024-06-26T08:45:25.8422226Z Deleted Containers: 2024-06-26T08:45:25.8422929Z 255a27a1a4efc331f3831ab6666c6f866b45de4a36bcfcc6f941d16be8e6baf9 2024-06-26T08:45:25.8423714Z 2024-06-26T08:45:29.7343172Z Deleted Images: 2024-06-26T08:45:29.7345021Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:45:29.7347297Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:45:29.7348968Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:45:29.7350086Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:45:29.7351045Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:45:29.7352081Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:45:29.7353177Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:45:29.7354084Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:45:29.7354970Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:45:29.7355994Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:45:29.7356822Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:45:29.7357632Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:45:29.7358480Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:45:29.7359311Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:45:29.7360126Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:45:29.7360942Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:45:29.7361768Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:45:29.7362708Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:45:29.7363539Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:45:29.7364407Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:45:29.7365237Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:45:29.7366070Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:45:29.7367009Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:45:29.7367822Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:45:29.7368622Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:45:29.7369434Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:45:29.7370270Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:45:29.7371107Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:45:29.7371947Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:45:29.7372778Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:45:29.7373596Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:45:29.7374456Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:45:29.7375245Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:45:29.7376060Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:45:29.7376877Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:45:29.7377686Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:45:29.7378497Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:45:29.7379313Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:45:29.7380124Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:45:29.7380935Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:45:29.7381751Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:45:29.7382566Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:45:29.7383588Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:45:29.7384411Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:45:29.7385203Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:45:29.7386074Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:45:29.7387051Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:45:29.7387887Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:45:29.7388701Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:45:29.7389505Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:45:29.7390314Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:45:29.7391118Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:45:29.7391939Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:45:29.7392759Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:45:29.7393582Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:45:29.7394450Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:45:29.7395267Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:45:29.7396077Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:45:29.7396872Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:45:29.7397753Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:45:29.7398561Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:45:29.7399373Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:45:29.7400199Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:45:29.7401007Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:45:29.7401817Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:45:29.7402716Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:45:29.7403512Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:45:29.7404351Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:45:29.7405171Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:45:29.7405973Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:45:29.7406779Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:45:29.7407597Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:45:29.7408387Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:45:29.7409191Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:45:29.7410010Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:45:29.7410818Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:45:29.7411633Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:45:29.7412464Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:45:29.7413303Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:45:29.7414160Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:45:29.7414999Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:45:29.7415494Z 2024-06-26T08:45:29.7415645Z Total reclaimed space: 41.17GB 2024-06-26T08:45:29.7492183Z Post job cleanup. 2024-06-26T08:45:29.7540888Z Post job cleanup. 2024-06-26T08:45:29.8357647Z [command]/usr/bin/git version 2024-06-26T08:45:29.8399057Z git version 2.40.1 2024-06-26T08:45:29.8438940Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/71198adc-4e16-489b-a953-f01855945527' before making global git config changes 2024-06-26T08:45:29.8440407Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:45:29.8444759Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:45:29.8477553Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:45:29.8505642Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:45:29.8730318Z Entering 'android/libs/fbjni' 2024-06-26T08:45:29.8768351Z Entering 'third_party/FP16' 2024-06-26T08:45:29.8805630Z Entering 'third_party/FXdiv' 2024-06-26T08:45:29.8840651Z Entering 'third_party/NNPACK' 2024-06-26T08:45:29.8878695Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:45:29.8916424Z Entering 'third_party/XNNPACK' 2024-06-26T08:45:29.8969081Z Entering 'third_party/benchmark' 2024-06-26T08:45:29.9006013Z Entering 'third_party/cpp-httplib' 2024-06-26T08:45:29.9042061Z Entering 'third_party/cpuinfo' 2024-06-26T08:45:29.9078967Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:45:29.9115882Z Entering 'third_party/cutlass' 2024-06-26T08:45:29.9158517Z Entering 'third_party/eigen' 2024-06-26T08:45:29.9192556Z Entering 'third_party/fbgemm' 2024-06-26T08:45:29.9228582Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:45:29.9262633Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:45:29.9296775Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:45:29.9336244Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:45:29.9370785Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:45:29.9407180Z Entering 'third_party/flatbuffers' 2024-06-26T08:45:29.9446208Z Entering 'third_party/fmt' 2024-06-26T08:45:29.9482371Z Entering 'third_party/foxi' 2024-06-26T08:45:29.9518653Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:45:29.9554881Z Entering 'third_party/gloo' 2024-06-26T08:45:29.9589455Z Entering 'third_party/googletest' 2024-06-26T08:45:29.9624519Z Entering 'third_party/ideep' 2024-06-26T08:45:29.9658529Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:45:29.9697597Z Entering 'third_party/ittapi' 2024-06-26T08:45:29.9735139Z Entering 'third_party/kineto' 2024-06-26T08:45:29.9770137Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:45:29.9805304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:45:29.9842081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:45:29.9876280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:45:29.9910380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:45:29.9944020Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:45:29.9979234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:45:30.0015007Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:45:30.0051148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:45:30.0085534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:45:30.0122318Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:45:30.0154944Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:45:30.0191292Z Entering 'third_party/mimalloc' 2024-06-26T08:45:30.0227560Z Entering 'third_party/nccl/nccl' 2024-06-26T08:45:30.0262205Z Entering 'third_party/nlohmann' 2024-06-26T08:45:30.0294211Z Entering 'third_party/onnx' 2024-06-26T08:45:30.0342664Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:45:30.0377816Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:45:30.0414884Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:45:30.0452807Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:45:30.0488427Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:45:30.0522168Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:45:30.0552897Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:45:30.0587252Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:45:30.0620755Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:45:30.0653073Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:45:30.0687620Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:45:30.0721800Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:45:30.0757209Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:45:30.0807585Z Entering 'third_party/pocketfft' 2024-06-26T08:45:30.0844752Z Entering 'third_party/protobuf' 2024-06-26T08:45:30.0880230Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:45:30.0913502Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:45:30.0949924Z Entering 'third_party/psimd' 2024-06-26T08:45:30.0983252Z Entering 'third_party/pthreadpool' 2024-06-26T08:45:30.1017557Z Entering 'third_party/pybind11' 2024-06-26T08:45:30.1053502Z Entering 'third_party/python-peachpy' 2024-06-26T08:45:30.1090091Z Entering 'third_party/sleef' 2024-06-26T08:45:30.1125272Z Entering 'third_party/tensorpipe' 2024-06-26T08:45:30.1161935Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:45:30.1194135Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:45:30.1227831Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:45:30.1261234Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:45:30.1295946Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:45:30.1347913Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:45:30.1368162Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1376233Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:45:30.1405657Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:45:30.1616120Z Entering 'android/libs/fbjni' 2024-06-26T08:45:30.1637858Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1662165Z Entering 'third_party/FP16' 2024-06-26T08:45:30.1684383Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1707704Z Entering 'third_party/FXdiv' 2024-06-26T08:45:30.1729736Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1755502Z Entering 'third_party/NNPACK' 2024-06-26T08:45:30.1777713Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1802421Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:45:30.1822916Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1846404Z Entering 'third_party/XNNPACK' 2024-06-26T08:45:30.1868659Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1902912Z Entering 'third_party/benchmark' 2024-06-26T08:45:30.1926324Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1950365Z Entering 'third_party/cpp-httplib' 2024-06-26T08:45:30.1972284Z http.https://github.com/.extraheader 2024-06-26T08:45:30.1995770Z Entering 'third_party/cpuinfo' 2024-06-26T08:45:30.2018198Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2043236Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:45:30.2065543Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2089147Z Entering 'third_party/cutlass' 2024-06-26T08:45:30.2110891Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2139731Z Entering 'third_party/eigen' 2024-06-26T08:45:30.2161727Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2187886Z Entering 'third_party/fbgemm' 2024-06-26T08:45:30.2209852Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2233355Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:45:30.2254698Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2277810Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:45:30.2298668Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2321055Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:45:30.2341847Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2369213Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:45:30.2390389Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2412884Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:45:30.2433372Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2457563Z Entering 'third_party/flatbuffers' 2024-06-26T08:45:30.2480182Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2505725Z Entering 'third_party/fmt' 2024-06-26T08:45:30.2528013Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2552522Z Entering 'third_party/foxi' 2024-06-26T08:45:30.2574540Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2598216Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:45:30.2620071Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2643095Z Entering 'third_party/gloo' 2024-06-26T08:45:30.2664616Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2688405Z Entering 'third_party/googletest' 2024-06-26T08:45:30.2709202Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2733675Z Entering 'third_party/ideep' 2024-06-26T08:45:30.2755139Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2776628Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:45:30.2797906Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2827887Z Entering 'third_party/ittapi' 2024-06-26T08:45:30.2848892Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2871264Z Entering 'third_party/kineto' 2024-06-26T08:45:30.2891302Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2912357Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:45:30.2933353Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2955402Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:45:30.2975359Z http.https://github.com/.extraheader 2024-06-26T08:45:30.2998332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:45:30.3018319Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3040133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:45:30.3059588Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3080912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:45:30.3102919Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3125351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:45:30.3145751Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3170209Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:45:30.3191879Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3216301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:45:30.3236372Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3257354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:45:30.3278479Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3299748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:45:30.3319711Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3342950Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:45:30.3363436Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3386103Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:45:30.3405719Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3428909Z Entering 'third_party/mimalloc' 2024-06-26T08:45:30.3449106Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3470615Z Entering 'third_party/nccl/nccl' 2024-06-26T08:45:30.3490978Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3512536Z Entering 'third_party/nlohmann' 2024-06-26T08:45:30.3532792Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3554877Z Entering 'third_party/onnx' 2024-06-26T08:45:30.3575754Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3608688Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:45:30.3630036Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3651977Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:45:30.3672655Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3695363Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:45:30.3715782Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3738580Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:45:30.3759128Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3780030Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:45:30.3800607Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3822901Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:45:30.3843922Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3868782Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:45:30.3889666Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3915226Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:45:30.3936238Z http.https://github.com/.extraheader 2024-06-26T08:45:30.3960884Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:45:30.3983062Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4006121Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:45:30.4026592Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4048714Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:45:30.4070084Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4092794Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:45:30.4114038Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4137125Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:45:30.4157598Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4194753Z Entering 'third_party/pocketfft' 2024-06-26T08:45:30.4215848Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4238103Z Entering 'third_party/protobuf' 2024-06-26T08:45:30.4260569Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4284825Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:45:30.4307627Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4331246Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:45:30.4354237Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4378014Z Entering 'third_party/psimd' 2024-06-26T08:45:30.4399899Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4422925Z Entering 'third_party/pthreadpool' 2024-06-26T08:45:30.4443269Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4464334Z Entering 'third_party/pybind11' 2024-06-26T08:45:30.4485922Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4509330Z Entering 'third_party/python-peachpy' 2024-06-26T08:45:30.4530593Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4554685Z Entering 'third_party/sleef' 2024-06-26T08:45:30.4576368Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4600297Z Entering 'third_party/tensorpipe' 2024-06-26T08:45:30.4621302Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4643813Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:45:30.4665958Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4688024Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:45:30.4708667Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4729737Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:45:30.4750972Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4773486Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:45:30.4795287Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4816391Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:45:30.4838162Z http.https://github.com/.extraheader 2024-06-26T08:45:30.4936170Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:45:30.4953405Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:45:30.4959992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:45:30.4960514Z ##[endgroup] 2024-06-26T08:45:31.3705499Z Cleaning up orphan processes