2024-08-20T22:04:51.0334963Z Current runner version: '2.319.1' 2024-08-20T22:04:51.0341909Z Runner name: 'i-0fa2b5ff9d3cb2975' 2024-08-20T22:04:51.0342814Z Runner group name: 'Default' 2024-08-20T22:04:51.0343678Z Machine name: 'ip-10-0-21-76' 2024-08-20T22:04:51.0360470Z Testing runner upgrade compatibility 2024-08-20T22:04:51.1778185Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:04:51.1780605Z Actions: read 2024-08-20T22:04:51.1781386Z Attestations: read 2024-08-20T22:04:51.1781915Z Checks: read 2024-08-20T22:04:51.1782411Z Contents: read 2024-08-20T22:04:51.1782978Z Deployments: read 2024-08-20T22:04:51.1783766Z Discussions: read 2024-08-20T22:04:51.1784263Z Issues: read 2024-08-20T22:04:51.1784839Z Metadata: read 2024-08-20T22:04:51.1785348Z Packages: read 2024-08-20T22:04:51.1785812Z Pages: read 2024-08-20T22:04:51.1786377Z PullRequests: read 2024-08-20T22:04:51.1786937Z RepositoryProjects: read 2024-08-20T22:04:51.1787619Z SecurityEvents: read 2024-08-20T22:04:51.1788210Z Statuses: read 2024-08-20T22:04:51.1788984Z ##[endgroup] 2024-08-20T22:04:51.1792629Z Secret source: Actions 2024-08-20T22:04:51.1793485Z Prepare workflow directory 2024-08-20T22:04:51.3160366Z Prepare all required actions 2024-08-20T22:04:51.3335779Z Getting action download info 2024-08-20T22:04:51.5090382Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:04:52.1014424Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:04:56.0928453Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:04:56.2819319Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:04:56.6094422Z Getting action download info 2024-08-20T22:04:56.7134461Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:04:56.8844547Z Getting action download info 2024-08-20T22:04:56.9799951Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:04:57.1156639Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:04:57.1158785Z ##[group] Inputs 2024-08-20T22:04:57.1159268Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:04:57.1166287Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:04:57.1174045Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:04:57.1175204Z sync-tag: 2024-08-20T22:04:57.1176033Z timeout-minutes: 240 2024-08-20T22:04:57.1176379Z use-gha: 2024-08-20T22:04:57.1176671Z dashboard-tag: 2024-08-20T22:04:57.1177011Z s3-bucket: gha-artifacts 2024-08-20T22:04:57.1177364Z aws-role-to-assume: 2024-08-20T22:04:57.1177698Z ##[endgroup] 2024-08-20T22:04:57.1178763Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:04:57.1850573Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:04:57.1989983Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:04:57.2001834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:04:57.2002373Z ##[endgroup] 2024-08-20T22:04:58.5906642Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:04:58.5907174Z Instance Type: g5.4xlarge 2024-08-20T22:04:58.5908026Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:04:58.5908574Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:05:04.4192001Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:05:04.4192549Z with: 2024-08-20T22:05:04.4193283Z github-secret: *** 2024-08-20T22:05:04.4194222Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:04.4195212Z activate-with-label: false 2024-08-20T22:05:04.4195579Z label: with-ssh 2024-08-20T22:05:04.4195904Z remove-existing-keys: true 2024-08-20T22:05:04.4196263Z fail-silently: true 2024-08-20T22:05:04.4196576Z env: 2024-08-20T22:05:04.4196853Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:04.4197214Z ##[endgroup] 2024-08-20T22:05:04.5094709Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:05:04.5098194Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:05:04.7863356Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:05:04.8553703Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:05:04.8554324Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:05:04.9404577Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:05:04.9420159Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:05:04.9445166Z Login using: ssh ec2-user@ec2-34-201-1-145.compute-1.amazonaws.com 2024-08-20T22:05:04.9445926Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:05:04.9446734Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:04.9579608Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:05:04.9580168Z with: 2024-08-20T22:05:04.9580450Z submodules: recursive 2024-08-20T22:05:04.9580781Z fetch-depth: 0 2024-08-20T22:05:04.9581075Z env: 2024-08-20T22:05:04.9581355Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:04.9581690Z ##[endgroup] 2024-08-20T22:05:04.9772095Z ##[group]Run retry () { 2024-08-20T22:05:04.9772483Z retry () { 2024-08-20T22:05:04.9773002Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:05:04.9773560Z } 2024-08-20T22:05:04.9773879Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:05:04.9774316Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:05:04.9774806Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:04.9775265Z else 2024-08-20T22:05:04.9775614Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:04.9776046Z fi 2024-08-20T22:05:04.9776598Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:05:04.9786402Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:04.9786908Z env: 2024-08-20T22:05:04.9787192Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:04.9787612Z NO_SUDO: 2024-08-20T22:05:04.9787896Z ##[endgroup] 2024-08-20T22:05:04.9819695Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:05.1215805Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:05:05.1216245Z with: 2024-08-20T22:05:05.1216592Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:05.1217034Z fetch-depth: 0 2024-08-20T22:05:05.1217356Z submodules: recursive 2024-08-20T22:05:05.1217693Z quiet-checkout: true 2024-08-20T22:05:05.1218059Z repository: pytorch/pytorch 2024-08-20T22:05:05.1218550Z token: *** 2024-08-20T22:05:05.1218851Z ssh-strict: true 2024-08-20T22:05:05.1219176Z persist-credentials: true 2024-08-20T22:05:05.1219539Z clean: true 2024-08-20T22:05:05.1219900Z sparse-checkout-cone-mode: true 2024-08-20T22:05:05.1220295Z lfs: false 2024-08-20T22:05:05.1220597Z set-safe-directory: true 2024-08-20T22:05:05.1220942Z env: 2024-08-20T22:05:05.1221223Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:05.1221560Z ##[endgroup] 2024-08-20T22:05:05.2212788Z Syncing repository: pytorch/pytorch 2024-08-20T22:05:05.2214300Z ##[group]Getting Git version info 2024-08-20T22:05:05.2215028Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:05.2215871Z [command]/usr/bin/git version 2024-08-20T22:05:05.2216236Z git version 2.40.1 2024-08-20T22:05:05.2222992Z ##[endgroup] 2024-08-20T22:05:05.2239451Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/907f9d59-0863-4e54-9ee3-2c8d8a19568c/.gitconfig' 2024-08-20T22:05:05.2244871Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/907f9d59-0863-4e54-9ee3-2c8d8a19568c' before making global git config changes 2024-08-20T22:05:05.2246097Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:05:05.2249901Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:05.2314238Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:05.2317952Z ##[group]Initializing the repository 2024-08-20T22:05:05.2321032Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:05.2368994Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:05:05.2370080Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:05:05.2370905Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:05:05.2371676Z hint: 2024-08-20T22:05:05.2372153Z hint: git config --global init.defaultBranch 2024-08-20T22:05:05.2372630Z hint: 2024-08-20T22:05:05.2373455Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:05:05.2374455Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:05:05.2375039Z hint: 2024-08-20T22:05:05.2375376Z hint: git branch -m 2024-08-20T22:05:05.2376122Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:05:05.2382323Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:05:05.2425953Z ##[endgroup] 2024-08-20T22:05:05.2426596Z ##[group]Disabling automatic garbage collection 2024-08-20T22:05:05.2429221Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:05:05.2472652Z ##[endgroup] 2024-08-20T22:05:05.2473240Z ##[group]Setting up auth 2024-08-20T22:05:05.2478355Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:05:05.2522359Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:05:05.2882531Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:05:05.2928138Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:05:05.3287059Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:05:05.3346890Z ##[endgroup] 2024-08-20T22:05:05.3347787Z ##[group]Fetching the repository 2024-08-20T22:05:05.3353773Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:05:08.1620016Z remote: Enumerating objects: 1008518 2024-08-20T22:05:08.1620578Z remote: Enumerating objects: 1010208, done. 2024-08-20T22:05:08.1622607Z remote: Counting objects: 0% (1/1690) 2024-08-20T22:05:08.1624433Z remote: Counting objects: 1% (17/1690) 2024-08-20T22:05:08.1625371Z remote: Counting objects: 2% (34/1690) 2024-08-20T22:05:08.1626160Z remote: Counting objects: 3% (51/1690) 2024-08-20T22:05:08.1626744Z remote: Counting objects: 4% (68/1690) 2024-08-20T22:05:08.1627677Z remote: Counting objects: 5% (85/1690) 2024-08-20T22:05:08.1628348Z remote: Counting objects: 6% (102/1690) 2024-08-20T22:05:08.1628893Z remote: Counting objects: 7% (119/1690) 2024-08-20T22:05:08.1629786Z remote: Counting objects: 8% (136/1690) 2024-08-20T22:05:08.1630662Z remote: Counting objects: 9% (153/1690) 2024-08-20T22:05:08.1631607Z remote: Counting objects: 10% (169/1690) 2024-08-20T22:05:08.1632385Z remote: Counting objects: 11% (186/1690) 2024-08-20T22:05:08.1633183Z remote: Counting objects: 12% (203/1690) 2024-08-20T22:05:08.1633911Z remote: Counting objects: 13% (220/1690) 2024-08-20T22:05:08.1634789Z remote: Counting objects: 14% (237/1690) 2024-08-20T22:05:08.1635466Z remote: Counting objects: 15% (254/1690) 2024-08-20T22:05:08.1636131Z remote: Counting objects: 16% (271/1690) 2024-08-20T22:05:08.1637374Z remote: Counting objects: 17% (288/1690) 2024-08-20T22:05:08.1638044Z remote: Counting objects: 18% (305/1690) 2024-08-20T22:05:08.1638681Z remote: Counting objects: 19% (322/1690) 2024-08-20T22:05:08.1639861Z remote: Counting objects: 20% (338/1690) 2024-08-20T22:05:08.1640459Z remote: Counting objects: 21% (355/1690) 2024-08-20T22:05:08.1641293Z remote: Counting objects: 22% (372/1690) 2024-08-20T22:05:08.1641947Z remote: Counting objects: 23% (389/1690) 2024-08-20T22:05:08.1642590Z remote: Counting objects: 24% (406/1690) 2024-08-20T22:05:08.1643243Z remote: Counting objects: 25% (423/1690) 2024-08-20T22:05:08.1644191Z remote: Counting objects: 26% (440/1690) 2024-08-20T22:05:08.1644745Z remote: Counting objects: 27% (457/1690) 2024-08-20T22:05:08.1645246Z remote: Counting objects: 28% (474/1690) 2024-08-20T22:05:08.1645738Z remote: Counting objects: 29% (491/1690) 2024-08-20T22:05:08.1646224Z remote: Counting objects: 30% (507/1690) 2024-08-20T22:05:08.1646720Z remote: Counting objects: 31% (524/1690) 2024-08-20T22:05:08.1647225Z remote: Counting objects: 32% (541/1690) 2024-08-20T22:05:08.1647728Z remote: Counting objects: 33% (558/1690) 2024-08-20T22:05:08.1648222Z remote: Counting objects: 34% (575/1690) 2024-08-20T22:05:08.1648720Z remote: Counting objects: 35% (592/1690) 2024-08-20T22:05:08.1649218Z remote: Counting objects: 36% (609/1690) 2024-08-20T22:05:08.1649713Z remote: Counting objects: 37% (626/1690) 2024-08-20T22:05:08.1650201Z remote: Counting objects: 38% (643/1690) 2024-08-20T22:05:08.1650880Z remote: Counting objects: 39% (660/1690) 2024-08-20T22:05:08.1651377Z remote: Counting objects: 40% (676/1690) 2024-08-20T22:05:08.1651878Z remote: Counting objects: 41% (693/1690) 2024-08-20T22:05:08.1652366Z remote: Counting objects: 42% (710/1690) 2024-08-20T22:05:08.1652859Z remote: Counting objects: 43% (727/1690) 2024-08-20T22:05:08.1653354Z remote: Counting objects: 44% (744/1690) 2024-08-20T22:05:08.1653849Z remote: Counting objects: 45% (761/1690) 2024-08-20T22:05:08.1654336Z remote: Counting objects: 46% (778/1690) 2024-08-20T22:05:08.1654834Z remote: Counting objects: 47% (795/1690) 2024-08-20T22:05:08.1655328Z remote: Counting objects: 48% (812/1690) 2024-08-20T22:05:08.1655829Z remote: Counting objects: 49% (829/1690) 2024-08-20T22:05:08.1656316Z remote: Counting objects: 50% (845/1690) 2024-08-20T22:05:08.1656819Z remote: Counting objects: 51% (862/1690) 2024-08-20T22:05:08.1657325Z remote: Counting objects: 52% (879/1690) 2024-08-20T22:05:08.1657816Z remote: Counting objects: 53% (896/1690) 2024-08-20T22:05:08.1658300Z remote: Counting objects: 54% (913/1690) 2024-08-20T22:05:08.1658796Z remote: Counting objects: 55% (930/1690) 2024-08-20T22:05:08.1659302Z remote: Counting objects: 56% (947/1690) 2024-08-20T22:05:08.1659804Z remote: Counting objects: 57% (964/1690) 2024-08-20T22:05:08.1660292Z remote: Counting objects: 58% (981/1690) 2024-08-20T22:05:08.1660799Z remote: Counting objects: 59% (998/1690) 2024-08-20T22:05:08.1661312Z remote: Counting objects: 60% (1014/1690) 2024-08-20T22:05:08.1661843Z remote: Counting objects: 61% (1031/1690) 2024-08-20T22:05:08.1662360Z remote: Counting objects: 62% (1048/1690) 2024-08-20T22:05:08.1662896Z remote: Counting objects: 63% (1065/1690) 2024-08-20T22:05:08.1663436Z remote: Counting objects: 64% (1082/1690) 2024-08-20T22:05:08.1663956Z remote: Counting objects: 65% (1099/1690) 2024-08-20T22:05:08.1664478Z remote: Counting objects: 66% (1116/1690) 2024-08-20T22:05:08.1664995Z remote: Counting objects: 67% (1133/1690) 2024-08-20T22:05:08.1665506Z remote: Counting objects: 68% (1150/1690) 2024-08-20T22:05:08.1666015Z remote: Counting objects: 69% (1167/1690) 2024-08-20T22:05:08.1666529Z remote: Counting objects: 70% (1183/1690) 2024-08-20T22:05:08.1667039Z remote: Counting objects: 71% (1200/1690) 2024-08-20T22:05:08.1667648Z remote: Counting objects: 72% (1217/1690) 2024-08-20T22:05:08.1668156Z remote: Counting objects: 73% (1234/1690) 2024-08-20T22:05:08.1668658Z remote: Counting objects: 74% (1251/1690) 2024-08-20T22:05:08.1669157Z remote: Counting objects: 75% (1268/1690) 2024-08-20T22:05:08.1669662Z remote: Counting objects: 76% (1285/1690) 2024-08-20T22:05:08.1670264Z remote: Counting objects: 77% (1302/1690) 2024-08-20T22:05:08.1670777Z remote: Counting objects: 78% (1319/1690) 2024-08-20T22:05:08.1671275Z remote: Counting objects: 79% (1336/1690) 2024-08-20T22:05:08.1671785Z remote: Counting objects: 80% (1352/1690) 2024-08-20T22:05:08.1672292Z remote: Counting objects: 81% (1369/1690) 2024-08-20T22:05:08.1672805Z remote: Counting objects: 82% (1386/1690) 2024-08-20T22:05:08.1673312Z remote: Counting objects: 83% (1403/1690) 2024-08-20T22:05:08.1673827Z remote: Counting objects: 84% (1420/1690) 2024-08-20T22:05:08.1674383Z remote: Counting objects: 85% (1437/1690) 2024-08-20T22:05:08.1674903Z remote: Counting objects: 86% (1454/1690) 2024-08-20T22:05:08.1675399Z remote: Counting objects: 87% (1471/1690) 2024-08-20T22:05:08.1675911Z remote: Counting objects: 88% (1488/1690) 2024-08-20T22:05:08.1676421Z remote: Counting objects: 89% (1505/1690) 2024-08-20T22:05:08.1677024Z remote: Counting objects: 90% (1521/1690) 2024-08-20T22:05:08.1677531Z remote: Counting objects: 91% (1538/1690) 2024-08-20T22:05:08.1678038Z remote: Counting objects: 92% (1555/1690) 2024-08-20T22:05:08.1678558Z remote: Counting objects: 93% (1572/1690) 2024-08-20T22:05:08.1679078Z remote: Counting objects: 94% (1589/1690) 2024-08-20T22:05:08.1679586Z remote: Counting objects: 95% (1606/1690) 2024-08-20T22:05:08.1680099Z remote: Counting objects: 96% (1623/1690) 2024-08-20T22:05:08.1680606Z remote: Counting objects: 97% (1640/1690) 2024-08-20T22:05:08.1681122Z remote: Counting objects: 98% (1657/1690) 2024-08-20T22:05:08.1681637Z remote: Counting objects: 99% (1674/1690) 2024-08-20T22:05:08.1682158Z remote: Counting objects: 100% (1690/1690) 2024-08-20T22:05:08.1682706Z remote: Counting objects: 100% (1690/1690), done. 2024-08-20T22:05:08.1742212Z remote: Compressing objects: 0% (1/875) 2024-08-20T22:05:08.2128645Z remote: Compressing objects: 1% (9/875) 2024-08-20T22:05:08.2406930Z remote: Compressing objects: 2% (18/875) 2024-08-20T22:05:08.2869092Z remote: Compressing objects: 3% (27/875) 2024-08-20T22:05:08.3551785Z remote: Compressing objects: 4% (35/875) 2024-08-20T22:05:08.4099660Z remote: Compressing objects: 5% (44/875) 2024-08-20T22:05:08.4583072Z remote: Compressing objects: 6% (53/875) 2024-08-20T22:05:08.5062584Z remote: Compressing objects: 7% (62/875) 2024-08-20T22:05:08.5525010Z remote: Compressing objects: 8% (70/875) 2024-08-20T22:05:08.5831144Z remote: Compressing objects: 9% (79/875) 2024-08-20T22:05:08.6063101Z remote: Compressing objects: 10% (88/875) 2024-08-20T22:05:08.6367624Z remote: Compressing objects: 11% (97/875) 2024-08-20T22:05:08.6585974Z remote: Compressing objects: 12% (105/875) 2024-08-20T22:05:08.6779904Z remote: Compressing objects: 13% (114/875) 2024-08-20T22:05:08.6927423Z remote: Compressing objects: 14% (123/875) 2024-08-20T22:05:08.7052261Z remote: Compressing objects: 15% (132/875) 2024-08-20T22:05:08.7142731Z remote: Compressing objects: 16% (140/875) 2024-08-20T22:05:08.7244519Z remote: Compressing objects: 17% (149/875) 2024-08-20T22:05:08.7313096Z remote: Compressing objects: 18% (158/875) 2024-08-20T22:05:08.7363945Z remote: Compressing objects: 19% (167/875) 2024-08-20T22:05:08.7412838Z remote: Compressing objects: 20% (175/875) 2024-08-20T22:05:08.7444219Z remote: Compressing objects: 21% (184/875) 2024-08-20T22:05:08.7465452Z remote: Compressing objects: 22% (193/875) 2024-08-20T22:05:08.7471317Z remote: Compressing objects: 23% (202/875) 2024-08-20T22:05:08.7479259Z remote: Compressing objects: 24% (210/875) 2024-08-20T22:05:08.7489697Z remote: Compressing objects: 25% (219/875) 2024-08-20T22:05:08.7498288Z remote: Compressing objects: 26% (228/875) 2024-08-20T22:05:08.7506902Z remote: Compressing objects: 27% (237/875) 2024-08-20T22:05:08.7530765Z remote: Compressing objects: 28% (245/875) 2024-08-20T22:05:08.7545492Z remote: Compressing objects: 29% (254/875) 2024-08-20T22:05:08.7563082Z remote: Compressing objects: 30% (263/875) 2024-08-20T22:05:08.7568159Z remote: Compressing objects: 31% (272/875) 2024-08-20T22:05:08.7581006Z remote: Compressing objects: 32% (280/875) 2024-08-20T22:05:08.7588790Z remote: Compressing objects: 33% (289/875) 2024-08-20T22:05:08.7597699Z remote: Compressing objects: 34% (298/875) 2024-08-20T22:05:08.7603457Z remote: Compressing objects: 35% (307/875) 2024-08-20T22:05:08.7618970Z remote: Compressing objects: 36% (315/875) 2024-08-20T22:05:08.7629048Z remote: Compressing objects: 37% (324/875) 2024-08-20T22:05:08.7636489Z remote: Compressing objects: 38% (333/875) 2024-08-20T22:05:08.7643714Z remote: Compressing objects: 39% (342/875) 2024-08-20T22:05:08.7648212Z remote: Compressing objects: 40% (350/875) 2024-08-20T22:05:08.7657174Z remote: Compressing objects: 41% (359/875) 2024-08-20T22:05:08.7662189Z remote: Compressing objects: 42% (368/875) 2024-08-20T22:05:08.7668222Z remote: Compressing objects: 43% (377/875) 2024-08-20T22:05:08.7676417Z remote: Compressing objects: 44% (385/875) 2024-08-20T22:05:08.7682638Z remote: Compressing objects: 45% (394/875) 2024-08-20T22:05:08.7687371Z remote: Compressing objects: 46% (403/875) 2024-08-20T22:05:08.7690625Z remote: Compressing objects: 47% (412/875) 2024-08-20T22:05:08.7695910Z remote: Compressing objects: 48% (420/875) 2024-08-20T22:05:08.7699058Z remote: Compressing objects: 49% (429/875) 2024-08-20T22:05:08.7702598Z remote: Compressing objects: 50% (438/875) 2024-08-20T22:05:08.7706282Z remote: Compressing objects: 51% (447/875) 2024-08-20T22:05:08.7709082Z remote: Compressing objects: 52% (455/875) 2024-08-20T22:05:08.7712490Z remote: Compressing objects: 53% (464/875) 2024-08-20T22:05:08.7715598Z remote: Compressing objects: 54% (473/875) 2024-08-20T22:05:08.7717985Z remote: Compressing objects: 55% (482/875) 2024-08-20T22:05:08.7719667Z remote: Compressing objects: 56% (490/875) 2024-08-20T22:05:08.7721232Z remote: Compressing objects: 57% (499/875) 2024-08-20T22:05:08.7722541Z remote: Compressing objects: 58% (508/875) 2024-08-20T22:05:08.7724434Z remote: Compressing objects: 59% (517/875) 2024-08-20T22:05:08.7725578Z remote: Compressing objects: 60% (525/875) 2024-08-20T22:05:08.7726211Z remote: Compressing objects: 61% (534/875) 2024-08-20T22:05:08.7726745Z remote: Compressing objects: 62% (543/875) 2024-08-20T22:05:08.7727273Z remote: Compressing objects: 63% (552/875) 2024-08-20T22:05:08.7732227Z remote: Compressing objects: 64% (560/875) 2024-08-20T22:05:08.7740299Z remote: Compressing objects: 65% (569/875) 2024-08-20T22:05:08.7746479Z remote: Compressing objects: 66% (578/875) 2024-08-20T22:05:08.7750645Z remote: Compressing objects: 67% (587/875) 2024-08-20T22:05:08.7755917Z remote: Compressing objects: 68% (595/875) 2024-08-20T22:05:08.7760618Z remote: Compressing objects: 69% (604/875) 2024-08-20T22:05:08.7765305Z remote: Compressing objects: 70% (613/875) 2024-08-20T22:05:08.7768311Z remote: Compressing objects: 71% (622/875) 2024-08-20T22:05:08.7773183Z remote: Compressing objects: 72% (630/875) 2024-08-20T22:05:08.7778919Z remote: Compressing objects: 73% (639/875) 2024-08-20T22:05:08.7782711Z remote: Compressing objects: 74% (648/875) 2024-08-20T22:05:08.7786753Z remote: Compressing objects: 75% (657/875) 2024-08-20T22:05:08.7792047Z remote: Compressing objects: 76% (665/875) 2024-08-20T22:05:08.7797835Z remote: Compressing objects: 77% (674/875) 2024-08-20T22:05:08.7798912Z remote: Compressing objects: 78% (683/875) 2024-08-20T22:05:08.7802251Z remote: Compressing objects: 79% (692/875) 2024-08-20T22:05:08.7805385Z remote: Compressing objects: 80% (700/875) 2024-08-20T22:05:08.7808457Z remote: Compressing objects: 81% (709/875) 2024-08-20T22:05:08.7812477Z remote: Compressing objects: 82% (718/875) 2024-08-20T22:05:08.7814361Z remote: Compressing objects: 83% (727/875) 2024-08-20T22:05:08.7817777Z remote: Compressing objects: 84% (735/875) 2024-08-20T22:05:08.7820010Z remote: Compressing objects: 85% (744/875) 2024-08-20T22:05:08.7823696Z remote: Compressing objects: 86% (753/875) 2024-08-20T22:05:08.7826307Z remote: Compressing objects: 87% (762/875) 2024-08-20T22:05:08.7830294Z remote: Compressing objects: 88% (770/875) 2024-08-20T22:05:08.7833417Z remote: Compressing objects: 89% (779/875) 2024-08-20T22:05:08.7835899Z remote: Compressing objects: 90% (788/875) 2024-08-20T22:05:08.7836668Z remote: Compressing objects: 91% (797/875) 2024-08-20T22:05:08.7839044Z remote: Compressing objects: 92% (805/875) 2024-08-20T22:05:08.7839606Z remote: Compressing objects: 93% (814/875) 2024-08-20T22:05:08.7842864Z remote: Compressing objects: 94% (823/875) 2024-08-20T22:05:08.7844759Z remote: Compressing objects: 95% (832/875) 2024-08-20T22:05:08.7846855Z remote: Compressing objects: 96% (840/875) 2024-08-20T22:05:08.7848551Z remote: Compressing objects: 97% (849/875) 2024-08-20T22:05:08.7850805Z remote: Compressing objects: 98% (858/875) 2024-08-20T22:05:08.7853165Z remote: Compressing objects: 99% (867/875) 2024-08-20T22:05:08.7853702Z remote: Compressing objects: 100% (875/875) 2024-08-20T22:05:08.7854270Z remote: Compressing objects: 100% (875/875), done. 2024-08-20T22:05:31.3995224Z remote: Total 1010208 (delta 1086), reused 1298 (delta 812), pack-reused 1008518 (from 1) 2024-08-20T22:05:55.9432714Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:05:55.9471424Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:55.9476945Z ##[endgroup] 2024-08-20T22:05:55.9477493Z ##[group]Determining the checkout info 2024-08-20T22:05:55.9478369Z ##[endgroup] 2024-08-20T22:05:55.9480884Z ##[group]Checking out the ref 2024-08-20T22:05:55.9481823Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:57.6865505Z ##[endgroup] 2024-08-20T22:05:57.6866095Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:05:57.6871071Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:05:57.6929127Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:05:57.6970018Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:05:57.7010662Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:05:57.7050887Z ##[endgroup] 2024-08-20T22:05:57.7054035Z ##[group]Fetching submodules 2024-08-20T22:05:57.7054686Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:05:57.7437068Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:05:57.7804892Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:05:57.7806716Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:05:57.7810196Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:05:57.7814227Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:05:57.7817932Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:05:57.7821770Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:05:57.7825714Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:05:57.7830149Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:05:57.7834310Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:05:57.7838797Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:05:57.7842967Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:05:57.7847124Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:05:57.7851495Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:05:57.7856031Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:05:57.7860586Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:05:57.7865330Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:05:57.7874083Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:05:57.7878669Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:05:57.7883688Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:05:57.7888748Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:05:57.7893842Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:05:57.7898935Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:05:57.7904010Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:05:57.7909267Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:05:57.7914508Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:05:57.7920156Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:05:57.7925393Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:05:57.7930937Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:05:57.7936493Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:05:57.7942634Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:05:57.7948410Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:05:57.7954259Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:05:57.7959964Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:05:57.7969989Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:05:57.8008379Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:05:58.1164857Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:05:58.2867558Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:05:58.5331704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:05:58.7669278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:06:00.6672649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:06:11.9224155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:06:12.3391783Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:06:12.7800300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:06:13.3800057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:06:14.6184845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:06:16.7373817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:06:22.7544542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:06:24.1902422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:06:25.9723618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:06:27.3217763Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:06:27.7777553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:06:28.1146863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:06:29.4141277Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:06:29.7622403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:06:30.0154913Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:06:31.6065013Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:06:32.2947491Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:06:33.0397878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:06:39.0548230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:06:41.0406074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:06:45.3147445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:06:45.5292113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:06:54.7128819Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:06:54.8805850Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:06:55.0819968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:06:55.9968975Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:06:56.2789410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:06:57.0126286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:06:57.4471946Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:06:57.4621227Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:06:57.4740335Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:06:57.5047713Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:06:57.5503725Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:06:58.7359008Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:06:58.7645089Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:06:58.8163381Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:06:58.9275094Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:06:58.9680514Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:06:59.5938481Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:06:59.8813513Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:06:59.9757651Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:06:59.9778060Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:06:59.9781473Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:06:59.9784982Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:06:59.9788209Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:06:59.9792091Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:06:59.9824373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:07:01.0217506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:07:01.6168301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:07:03.7428924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:07:04.8013925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:07:05.1410558Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:07:05.2532556Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:07:05.7643720Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:07:05.8351959Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:07:05.8502650Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:07:06.0020004Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:07:06.0485876Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:07:06.0938727Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:07:06.1259817Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:07:06.1792604Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:07:06.1951488Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:07:06.1970948Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:07:06.2000914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:07:19.6700175Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:07:19.6917528Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:07:19.7895658Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:07:19.7915256Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:07:19.7918280Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:07:19.7922049Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:07:19.7953855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:07:20.3155088Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:07:21.5355056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:07:22.7345523Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:07:22.7364713Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:07:22.7368065Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:07:22.7371722Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:07:22.7375423Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:07:22.7379136Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:07:22.7383069Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:07:22.7386949Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:07:22.7391257Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:07:22.7423895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:07:23.5708668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:07:24.0150118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:07:25.2362743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:07:25.5182693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:07:26.0478727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:07:27.0752267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:07:34.0677401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:07:34.4731858Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:07:34.4969196Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:07:34.5402850Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:07:34.5570748Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:07:34.5590780Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:07:34.5625717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:07:34.8558241Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:07:34.8782529Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:07:34.9267314Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:07:35.0507609Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:07:35.0708264Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:07:35.1167718Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:07:35.1831749Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:07:35.2284797Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:07:35.2585826Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:07:35.3850056Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:07:35.8969522Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:07:35.9006768Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:07:35.9009931Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:07:35.9042568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:07:36.3134735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:07:37.2641799Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:07:37.3058632Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:07:37.3955899Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:07:37.3976713Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:07:37.3979930Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:07:37.3983271Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:07:37.3986792Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:07:37.3991269Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:07:37.3994846Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:07:37.3998515Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:07:37.4002108Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:07:37.4034833Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:07:37.8341220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:07:38.8964224Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:07:39.2365124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:07:45.3815869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:07:45.6361613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:07:45.8445732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:07:46.1255373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:07:52.8757332Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:07:52.9220359Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:07:52.9405443Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:07:53.0646126Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:07:53.0814405Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:07:53.1019266Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:07:53.1229618Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:07:53.1250596Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:07:53.1254066Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:07:53.1284630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:07:55.0324852Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:07:56.3727916Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:07:56.4253481Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:07:57.0731623Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:07:57.0873123Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:07:57.3972365Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:07:57.3998796Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:07:57.4002604Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:07:57.4034566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:07:57.8409507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:07:58.9168581Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:07:58.9994226Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:07:59.0112233Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:07:59.0261533Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:07:59.0690620Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:07:59.1028207Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:07:59.1514210Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:07:59.1858079Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:07:59.1878994Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:07:59.1882585Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:07:59.1885757Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:07:59.1889386Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:07:59.1920843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:08:00.2698372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:08:00.5089942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:08:01.6621291Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:08:02.6392962Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:08:02.6587269Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:08:02.7384384Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:08:02.7733429Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:08:02.7754051Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:02.7785960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:08:02.9844946Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:08:02.9890034Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:08:03.0253982Z Entering 'android/libs/fbjni' 2024-08-20T22:08:03.0306701Z Entering 'third_party/FP16' 2024-08-20T22:08:03.0360890Z Entering 'third_party/FXdiv' 2024-08-20T22:08:03.0412478Z Entering 'third_party/NNPACK' 2024-08-20T22:08:03.0464639Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:03.0516144Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:03.0584559Z Entering 'third_party/benchmark' 2024-08-20T22:08:03.0635516Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:03.0685985Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:03.0736487Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:03.0787408Z Entering 'third_party/cutlass' 2024-08-20T22:08:03.0846056Z Entering 'third_party/eigen' 2024-08-20T22:08:03.0899719Z Entering 'third_party/fbgemm' 2024-08-20T22:08:03.0952155Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:03.1003550Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:03.1054602Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:03.1111816Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:03.1162708Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:03.1215942Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:03.1270856Z Entering 'third_party/fmt' 2024-08-20T22:08:03.1322598Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:03.1379347Z Entering 'third_party/gloo' 2024-08-20T22:08:03.1431426Z Entering 'third_party/googletest' 2024-08-20T22:08:03.1485072Z Entering 'third_party/ideep' 2024-08-20T22:08:03.1534328Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:03.1598452Z Entering 'third_party/ittapi' 2024-08-20T22:08:03.1650321Z Entering 'third_party/kineto' 2024-08-20T22:08:03.1701348Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:03.1751783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:03.1805220Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:03.1856836Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:03.1909401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:03.1957343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:03.2011808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:03.2062677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:03.2113765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:03.2165890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:03.2220449Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:03.2273647Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:03.2327285Z Entering 'third_party/mimalloc' 2024-08-20T22:08:03.2379990Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:03.2431493Z Entering 'third_party/nlohmann' 2024-08-20T22:08:03.2484350Z Entering 'third_party/onnx' 2024-08-20T22:08:03.2549750Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:03.2600327Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:03.2656750Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:03.2709058Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:03.2762541Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:03.2813090Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:03.2863896Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:03.2914938Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:03.2965721Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:03.3020097Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:03.3069094Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:03.3121815Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:03.3175067Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:03.3248961Z Entering 'third_party/pocketfft' 2024-08-20T22:08:03.3298502Z Entering 'third_party/protobuf' 2024-08-20T22:08:03.3352387Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:03.3402575Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:03.3456612Z Entering 'third_party/psimd' 2024-08-20T22:08:03.3508344Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:03.3560467Z Entering 'third_party/pybind11' 2024-08-20T22:08:03.3612049Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:03.3663389Z Entering 'third_party/sleef' 2024-08-20T22:08:03.3714839Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:03.3765623Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:03.3815253Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:03.3865848Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:03.3916141Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:03.3964154Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:03.4039441Z ##[endgroup] 2024-08-20T22:08:03.4044038Z ##[group]Persisting credentials for submodules 2024-08-20T22:08:03.4046706Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:08:03.4414727Z Entering 'android/libs/fbjni' 2024-08-20T22:08:03.4481322Z Entering 'third_party/FP16' 2024-08-20T22:08:03.4549356Z Entering 'third_party/FXdiv' 2024-08-20T22:08:03.4615936Z Entering 'third_party/NNPACK' 2024-08-20T22:08:03.4683304Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:03.4750932Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:03.4833093Z Entering 'third_party/benchmark' 2024-08-20T22:08:03.4902963Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:03.4971281Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:03.5039718Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:03.5106734Z Entering 'third_party/cutlass' 2024-08-20T22:08:03.5181100Z Entering 'third_party/eigen' 2024-08-20T22:08:03.5253515Z Entering 'third_party/fbgemm' 2024-08-20T22:08:03.5319188Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:03.5387441Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:03.5459153Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:03.5531222Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:03.5597957Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:03.5667778Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:03.5737058Z Entering 'third_party/fmt' 2024-08-20T22:08:03.5804807Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:03.5872151Z Entering 'third_party/gloo' 2024-08-20T22:08:03.5939730Z Entering 'third_party/googletest' 2024-08-20T22:08:03.6007279Z Entering 'third_party/ideep' 2024-08-20T22:08:03.6072491Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:03.6147664Z Entering 'third_party/ittapi' 2024-08-20T22:08:03.6213786Z Entering 'third_party/kineto' 2024-08-20T22:08:03.6280913Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:03.6345631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:03.6413546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:03.6481702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:03.6549608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:03.6618895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:03.6692092Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:03.6760270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:03.6826855Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:03.6894407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:03.6964826Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:03.7034570Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:03.7104652Z Entering 'third_party/mimalloc' 2024-08-20T22:08:03.7172825Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:03.7243858Z Entering 'third_party/nlohmann' 2024-08-20T22:08:03.7311536Z Entering 'third_party/onnx' 2024-08-20T22:08:03.7393069Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:03.7460475Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:03.7533607Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:03.7599766Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:03.7665677Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:03.7731754Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:03.7798135Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:03.7866057Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:03.7936135Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:03.8003289Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:03.8067819Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:03.8135401Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:03.8206000Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:03.8295671Z Entering 'third_party/pocketfft' 2024-08-20T22:08:03.8364484Z Entering 'third_party/protobuf' 2024-08-20T22:08:03.8433518Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:03.8500943Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:03.8571238Z Entering 'third_party/psimd' 2024-08-20T22:08:03.8637915Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:03.8704947Z Entering 'third_party/pybind11' 2024-08-20T22:08:03.8772681Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:03.8839972Z Entering 'third_party/sleef' 2024-08-20T22:08:03.8911300Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:03.8977354Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:03.9043745Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:03.9109643Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:03.9175756Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:03.9238708Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:03.9335753Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:08:03.9700589Z Entering 'android/libs/fbjni' 2024-08-20T22:08:03.9761234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:08:03.9782443Z Entering 'third_party/FP16' 2024-08-20T22:08:03.9844994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:08:03.9865970Z Entering 'third_party/FXdiv' 2024-08-20T22:08:03.9929119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:08:03.9950856Z Entering 'third_party/NNPACK' 2024-08-20T22:08:04.0012407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:08:04.0032956Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:04.0094330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:08:04.0115633Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:04.0177483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:08:04.0215036Z Entering 'third_party/benchmark' 2024-08-20T22:08:04.0278161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:04.0298834Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:04.0359824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:08:04.0380488Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:04.0441259Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:04.0462644Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:04.0523492Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:08:04.0544934Z Entering 'third_party/cutlass' 2024-08-20T22:08:04.0605604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:04.0633551Z Entering 'third_party/eigen' 2024-08-20T22:08:04.0697671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:08:04.0721584Z Entering 'third_party/fbgemm' 2024-08-20T22:08:04.0784266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:08:04.0804642Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:04.0871075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:08:04.0892573Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:04.0953921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:04.0974387Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:04.1034450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:04.1061569Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:04.1122152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:04.1142555Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:04.1203442Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:08:04.1225771Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:04.1290170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:08:04.1313921Z Entering 'third_party/fmt' 2024-08-20T22:08:04.1378779Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:04.1401528Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:04.1463438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:08:04.1484785Z Entering 'third_party/gloo' 2024-08-20T22:08:04.1547465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:08:04.1568459Z Entering 'third_party/googletest' 2024-08-20T22:08:04.1631096Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:04.1651500Z Entering 'third_party/ideep' 2024-08-20T22:08:04.1713709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:08:04.1732326Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:04.1793658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:08:04.1823436Z Entering 'third_party/ittapi' 2024-08-20T22:08:04.1885125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:08:04.1905973Z Entering 'third_party/kineto' 2024-08-20T22:08:04.1968869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:08:04.1987885Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:04.2050334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:08:04.2069193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:04.2131030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:08:04.2154634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:04.2217015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:08:04.2237946Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:04.2301232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:04.2321512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:04.2385493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:08:04.2403917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:04.2467532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:08:04.2493336Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:04.2557623Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:08:04.2578591Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:04.2646265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:04.2666698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:04.2733610Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:08:04.2760391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:04.2822961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:08:04.2846181Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:04.2907288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:08:04.2928583Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:04.2992922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:08:04.3015735Z Entering 'third_party/mimalloc' 2024-08-20T22:08:04.3079247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:08:04.3100147Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:04.3162123Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:08:04.3182742Z Entering 'third_party/nlohmann' 2024-08-20T22:08:04.3243268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:08:04.3265242Z Entering 'third_party/onnx' 2024-08-20T22:08:04.3326459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:08:04.3361468Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:04.3423098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:04.3444006Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:04.3507082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:04.3533691Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:04.3596334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:08:04.3616959Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:04.3678986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:04.3699029Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:04.3759541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:04.3779692Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:04.3841358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:08:04.3861464Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:04.3921199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:08:04.3943083Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:04.4003851Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:08:04.4023027Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:04.4083795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:08:04.4103480Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:04.4164168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:08:04.4181983Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:04.4245148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:08:04.4266594Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:04.4330053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:08:04.4356694Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:04.4418172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:08:04.4461146Z Entering 'third_party/pocketfft' 2024-08-20T22:08:04.4522312Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:08:04.4543578Z Entering 'third_party/protobuf' 2024-08-20T22:08:04.4605107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:08:04.4627606Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:04.4689281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:04.4710074Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:04.4771059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:04.4794345Z Entering 'third_party/psimd' 2024-08-20T22:08:04.4856403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:08:04.4877014Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:04.4938214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:08:04.4962171Z Entering 'third_party/pybind11' 2024-08-20T22:08:04.5023057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:04.5044928Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:04.5106040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:08:04.5125818Z Entering 'third_party/sleef' 2024-08-20T22:08:04.5186754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:08:04.5207512Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:04.5271353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:08:04.5288882Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:04.5350589Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:04.5370209Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:04.5429349Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:08:04.5448620Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:04.5508971Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:08:04.5530027Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:04.5590600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:04.5608536Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:04.5671182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:08:04.6453511Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:08:04.6819347Z Entering 'android/libs/fbjni' 2024-08-20T22:08:04.6872564Z Entering 'third_party/FP16' 2024-08-20T22:08:04.6923580Z Entering 'third_party/FXdiv' 2024-08-20T22:08:04.6975273Z Entering 'third_party/NNPACK' 2024-08-20T22:08:04.7026629Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:04.7078391Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:04.7146796Z Entering 'third_party/benchmark' 2024-08-20T22:08:04.7196828Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:04.7248375Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:04.7299679Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:04.7351313Z Entering 'third_party/cutlass' 2024-08-20T22:08:04.7409357Z Entering 'third_party/eigen' 2024-08-20T22:08:04.7462952Z Entering 'third_party/fbgemm' 2024-08-20T22:08:04.7513477Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:04.7564474Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:04.7615265Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:04.7674051Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:04.7721935Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:04.7774747Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:04.7828876Z Entering 'third_party/fmt' 2024-08-20T22:08:04.7879739Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:04.7931099Z Entering 'third_party/gloo' 2024-08-20T22:08:04.7982173Z Entering 'third_party/googletest' 2024-08-20T22:08:04.8032931Z Entering 'third_party/ideep' 2024-08-20T22:08:04.8082195Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:04.8140211Z Entering 'third_party/ittapi' 2024-08-20T22:08:04.8191163Z Entering 'third_party/kineto' 2024-08-20T22:08:04.8244388Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:04.8294349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:04.8346289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:04.8396546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:04.8448333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:04.8496706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:04.8552643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:04.8602663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:04.8653173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:04.8703984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:04.8757234Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:04.8812277Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:04.8866127Z Entering 'third_party/mimalloc' 2024-08-20T22:08:04.8918272Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:04.8970253Z Entering 'third_party/nlohmann' 2024-08-20T22:08:04.9022430Z Entering 'third_party/onnx' 2024-08-20T22:08:04.9087312Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:04.9138788Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:04.9194873Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:04.9246428Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:04.9296595Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:04.9347466Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:04.9397683Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:04.9449357Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:04.9498895Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:04.9552958Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:04.9601676Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:04.9654987Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:04.9708469Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:04.9783547Z Entering 'third_party/pocketfft' 2024-08-20T22:08:04.9835423Z Entering 'third_party/protobuf' 2024-08-20T22:08:04.9889485Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:04.9942185Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:04.9995620Z Entering 'third_party/psimd' 2024-08-20T22:08:05.0046272Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:05.0096650Z Entering 'third_party/pybind11' 2024-08-20T22:08:05.0147899Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:05.0198364Z Entering 'third_party/sleef' 2024-08-20T22:08:05.0249806Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:05.0299573Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:05.0349416Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:05.0398547Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:05.0452722Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:05.0499742Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:05.0577743Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:08:05.0940881Z Entering 'android/libs/fbjni' 2024-08-20T22:08:05.0994673Z Entering 'third_party/FP16' 2024-08-20T22:08:05.1047105Z Entering 'third_party/FXdiv' 2024-08-20T22:08:05.1100147Z Entering 'third_party/NNPACK' 2024-08-20T22:08:05.1152304Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:05.1204278Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:05.1274786Z Entering 'third_party/benchmark' 2024-08-20T22:08:05.1328965Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:05.1381757Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:05.1434696Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:05.1487470Z Entering 'third_party/cutlass' 2024-08-20T22:08:05.1548182Z Entering 'third_party/eigen' 2024-08-20T22:08:05.1601913Z Entering 'third_party/fbgemm' 2024-08-20T22:08:05.1655704Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:05.1705737Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:05.1756547Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:05.1812898Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:05.1864573Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:05.1917109Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:05.1972265Z Entering 'third_party/fmt' 2024-08-20T22:08:05.2024760Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:05.2077162Z Entering 'third_party/gloo' 2024-08-20T22:08:05.2128010Z Entering 'third_party/googletest' 2024-08-20T22:08:05.2179463Z Entering 'third_party/ideep' 2024-08-20T22:08:05.2228483Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:05.2288628Z Entering 'third_party/ittapi' 2024-08-20T22:08:05.2339857Z Entering 'third_party/kineto' 2024-08-20T22:08:05.2389773Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:05.2444583Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:05.2496118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:05.2546291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:05.2596021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:05.2644316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:05.2698728Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:05.2752662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:05.2801972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:05.2852646Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:05.2905814Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:05.2961714Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:05.3014852Z Entering 'third_party/mimalloc' 2024-08-20T22:08:05.3066898Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:05.3117841Z Entering 'third_party/nlohmann' 2024-08-20T22:08:05.3171270Z Entering 'third_party/onnx' 2024-08-20T22:08:05.3235998Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:05.3286175Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:05.3348411Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:05.3399849Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:05.3450148Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:05.3500844Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:05.3551230Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:05.3602236Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:05.3652383Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:05.3702128Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:05.3753346Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:05.3804581Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:05.3857091Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:05.3932914Z Entering 'third_party/pocketfft' 2024-08-20T22:08:05.3988658Z Entering 'third_party/protobuf' 2024-08-20T22:08:05.4062674Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:05.4113029Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:05.4167793Z Entering 'third_party/psimd' 2024-08-20T22:08:05.4219462Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:05.4270193Z Entering 'third_party/pybind11' 2024-08-20T22:08:05.4320978Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:05.4372353Z Entering 'third_party/sleef' 2024-08-20T22:08:05.4423166Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:05.4473417Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:05.4522911Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:05.4573041Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:05.4623256Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:05.4672971Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:05.4744951Z ##[endgroup] 2024-08-20T22:08:05.4795248Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:08:05.4832025Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:08:05.5044905Z Prepare all required actions 2024-08-20T22:08:05.5045410Z Getting action download info 2024-08-20T22:08:05.6800201Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:08:05.6800624Z env: 2024-08-20T22:08:05.6800901Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:05.6801240Z ##[endgroup] 2024-08-20T22:08:05.6860482Z ##[group]Run set -euo pipefail 2024-08-20T22:08:05.6860917Z set -euo pipefail 2024-08-20T22:08:05.6861306Z function get_ec2_metadata() { 2024-08-20T22:08:05.6861835Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:08:05.6862710Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:08:05.6863462Z  category=$1 2024-08-20T22:08:05.6863987Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:08:05.6864623Z  runner_name_str=i-0fa2b5ff9d3cb2975 2024-08-20T22:08:05.6865121Z  if [[ -f /.inarc ]]; then 2024-08-20T22:08:05.6865610Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:08:05.6866178Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:08:05.6866867Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:08:05.6867466Z  else 2024-08-20T22:08:05.6868052Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:08:05.6868645Z  fi 2024-08-20T22:08:05.6868933Z } 2024-08-20T22:08:05.6869289Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:08:05.6869871Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:08:05.6870519Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:08:05.6871082Z echo "system info $(uname -a)" 2024-08-20T22:08:05.6880134Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:05.6880625Z env: 2024-08-20T22:08:05.6880913Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:05.6881260Z ##[endgroup] 2024-08-20T22:08:05.6983073Z ami-id: ami-06c68f701d8090592 2024-08-20T22:08:05.7049000Z instance-id: i-0fa2b5ff9d3cb2975 2024-08-20T22:08:05.7106354Z instance-type: g5.4xlarge 2024-08-20T22:08:05.7120041Z system info Linux ip-10-0-21-76.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:08:05.7143446Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:05.7144333Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:05.7154155Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:05.7154643Z env: 2024-08-20T22:08:05.7154923Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:05.7155269Z ##[endgroup] 2024-08-20T22:08:05.7228674Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:08:05.7229280Z if systemctl is-active --quiet docker; then 2024-08-20T22:08:05.7229817Z  echo "Docker daemon is running..."; 2024-08-20T22:08:05.7230437Z else 2024-08-20T22:08:05.7230922Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:08:05.7231511Z fi 2024-08-20T22:08:05.7240142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:05.7240635Z env: 2024-08-20T22:08:05.7240910Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:05.7241259Z ##[endgroup] 2024-08-20T22:08:05.7335578Z Docker daemon is running... 2024-08-20T22:08:05.7384168Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:08:05.7384733Z with: 2024-08-20T22:08:05.7385007Z shell: bash 2024-08-20T22:08:05.7385301Z timeout_minutes: 5 2024-08-20T22:08:05.7385625Z max_attempts: 3 2024-08-20T22:08:05.7385941Z retry_wait_seconds: 30 2024-08-20T22:08:05.7389122Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:08:05.7392063Z polling_interval_seconds: 1 2024-08-20T22:08:05.7392433Z warning_on_retry: true 2024-08-20T22:08:05.7392785Z continue_on_error: false 2024-08-20T22:08:05.7393123Z env: 2024-08-20T22:08:05.7393396Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:05.7393743Z AWS_RETRY_MODE: standard 2024-08-20T22:08:05.7394090Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:08:05.7394434Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:08:05.7394809Z ##[endgroup] 2024-08-20T22:08:07.0793551Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:07.0794400Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:07.0795238Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:07.0795746Z 2024-08-20T22:08:07.0795872Z Login Succeeded 2024-08-20T22:08:07.7976393Z Command completed after 1 attempt(s). 2024-08-20T22:08:07.8038738Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:07.8039457Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:07.8040105Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:07.8050438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:07.8050939Z env: 2024-08-20T22:08:07.8051221Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:07.8051581Z ##[endgroup] 2024-08-20T22:08:07.8145963Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:07.8146728Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:07.8147324Z # shellcheck disable=SC2046 2024-08-20T22:08:07.8147878Z docker stop $(docker ps -q) || true 2024-08-20T22:08:07.8148399Z # Prune all of the docker images 2024-08-20T22:08:07.8148847Z docker system prune -af 2024-08-20T22:08:07.8157670Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:07.8158164Z env: 2024-08-20T22:08:07.8158443Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:07.8158794Z ##[endgroup] 2024-08-20T22:08:07.8480845Z "docker stop" requires at least 1 argument. 2024-08-20T22:08:07.8481751Z See 'docker stop --help'. 2024-08-20T22:08:07.8482056Z 2024-08-20T22:08:07.8482355Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:08:07.8482847Z 2024-08-20T22:08:07.8483722Z Stop one or more running containers 2024-08-20T22:08:11.0146243Z Deleted Images: 2024-08-20T22:08:11.0147143Z untagged: pytorch/manylinux-builder:cuda12.4 2024-08-20T22:08:11.0148524Z untagged: pytorch/manylinux-builder@sha256:7725463652a1c2549bc65956a5bb994a17e810fddafd661178f5a6dbd219e8b2 2024-08-20T22:08:11.0187934Z deleted: sha256:efaee664cd962834357b8bf241b8c4feac79ea2dbc636135704bc64bc5724447 2024-08-20T22:08:11.0188792Z deleted: sha256:1cc9f3c0d46a56331445c8cd28ce10f621b35d27f4bfeff493224e8adec41a58 2024-08-20T22:08:11.0189674Z deleted: sha256:c123174a23f907c9db3a54db5239be691c3f34b24cae50c3a70c34d613759294 2024-08-20T22:08:11.0190491Z deleted: sha256:1331679934bc883dffe4cf503929eeddb81fb84096f49c6db28feb1f061b04ff 2024-08-20T22:08:11.0191326Z deleted: sha256:bcfc7bfd2e88b2a744955c7f4ab1f8d431f5cb3e0e8c14c86e9c26132175137d 2024-08-20T22:08:11.0192152Z deleted: sha256:87602668b0b8ca45c1313a4515b28d958cb631e42201dea028ae79844e7b39c0 2024-08-20T22:08:11.0192966Z deleted: sha256:b2d291b2b5b81a97ce4249bf487c2bdf701b4bab27b16d4532f43138c360d826 2024-08-20T22:08:11.0194034Z deleted: sha256:9a5c6ccc25a086f18b54de2c5e3c5ef8bf3db84376e2815202620fe4eb612da8 2024-08-20T22:08:11.0194861Z deleted: sha256:6dd1c3c49490597a5ceef26c087408451a1ec49f4e761beba349a9c22fe75a22 2024-08-20T22:08:11.0195673Z deleted: sha256:651a979337413f82708e36195f04bf8afc6ad530652f4536af008d59ef63fc1e 2024-08-20T22:08:11.0196469Z deleted: sha256:23f7691823044cb1b0e5e110856ebf01fbf8a06909f4a9ef5eacab45da2fe3f6 2024-08-20T22:08:11.0197265Z deleted: sha256:11491d1e279edd3633545d2f7d8a41f0e08d243b14591f63669c92ae0dc40202 2024-08-20T22:08:11.0198070Z deleted: sha256:667784a6cd7b2f98661a158a82541cc7aa0b6a6df1262c6c5811932051171585 2024-08-20T22:08:11.0198877Z deleted: sha256:91622dc153e8cdd1a7adc76725c18a01b4507a2cc80c0fc785d3f12a42c4bad3 2024-08-20T22:08:11.0199758Z deleted: sha256:0d1432dba2ae1fddaff582e8833cb735cd7d544de0761ff263d6a20c6e433c00 2024-08-20T22:08:11.0200560Z deleted: sha256:24d299370986b800148c865a0294eaa0ffb60e3dc8aa796037c15ba69b2f77e9 2024-08-20T22:08:11.0201368Z deleted: sha256:6365fced154bcf86d1175ca69ebd8984faa809817e25ad658fb154ca6cf94eb7 2024-08-20T22:08:11.0202189Z deleted: sha256:d5f96861f34d35b7c8e387fc5cb291dd437d350c8ff23c7ecc4ff42119c668ac 2024-08-20T22:08:11.0203005Z deleted: sha256:8ec1443495054baf1684da24b64c2a06b95db924f726734ddff109edd8faca20 2024-08-20T22:08:11.0203828Z deleted: sha256:8bece3790bad83ee529ccec08dd83d147ae1e01dad832f7df1d04a24aec4f1a2 2024-08-20T22:08:11.0204652Z deleted: sha256:a0624a0073a62e85607b4be4e4c2040d78d20c4ccb779de75bea1831e89a37da 2024-08-20T22:08:11.0205469Z deleted: sha256:cbb001ad780888e5a12efbc2d7583a2b828a4c94b744584e68ce7d1aef891967 2024-08-20T22:08:11.0206288Z deleted: sha256:ef737025fcf99e19b7cd389d4f4558bf3e9185dce6cf25d1088b2b2b6f11f970 2024-08-20T22:08:11.0207086Z deleted: sha256:016882d6d6c81f3b4399221a4a358a11a477a8441147b6552e55f44e23c2c6e1 2024-08-20T22:08:11.0207878Z deleted: sha256:1bfde69bb99337ec3c49c4231311c5f39d69ad7c6775d076f3d3f8a3afa93b4f 2024-08-20T22:08:11.0208709Z deleted: sha256:8caec6ba792e50111d3a0b54bbf67cf3cd106c1d3cc1bbf23f9668aa9724e7a5 2024-08-20T22:08:11.0209554Z deleted: sha256:d08c39913a3001e41697b0550ed7516e15825e43bdfa1c7f8d89f3d9a0524b6b 2024-08-20T22:08:11.0210385Z deleted: sha256:e17360b6ff03d3b73b31c356984176d27c83e3640b255a273534d0d648462032 2024-08-20T22:08:11.0211173Z deleted: sha256:2466196f153e7742867c5faaf84d14fadf812e2609093eebecbe4c6ad4ee0fd9 2024-08-20T22:08:11.0211984Z deleted: sha256:cf5a2ca15e073de5aff9a9591bb89af446f5c8d8539c33148c62d60db73ead37 2024-08-20T22:08:11.0212810Z deleted: sha256:1ac53e5f98c88af8b594781f12bbb416e15bf1a48cd0342d20c93b6538c06d04 2024-08-20T22:08:11.0213617Z deleted: sha256:66039151bed04d6596bb1850af7d5da1e7e5f2213f3b5af8e06202fbf9e79535 2024-08-20T22:08:11.0214423Z deleted: sha256:8cb15b5c841450240e4c96c9ca0a6ff7da76ee8875fa8dc94a70dd6d7e93d1f6 2024-08-20T22:08:11.0215246Z deleted: sha256:af6c56f31add2d54a56e29693cc949e758f12ec348e3a44cc54fe8ee79d16eba 2024-08-20T22:08:11.0216057Z deleted: sha256:d5d341c3c243770ea9b70c4674439e91ebcfa4b73e26ce67e0469718ea770ffa 2024-08-20T22:08:11.0216869Z deleted: sha256:0f54ac065eac806c885fa3294cd1792d17f7956abb7720ba68d98ec89c9c7dc0 2024-08-20T22:08:11.0217684Z deleted: sha256:6bf3daa861b1fd1361da0b38250b9133760235fd48c27fefff94dbae487cb41f 2024-08-20T22:08:11.0218583Z deleted: sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02 2024-08-20T22:08:11.0219062Z 2024-08-20T22:08:11.1190712Z Total reclaimed space: 11.24GB 2024-08-20T22:08:11.1237860Z ##[group]Run set +e 2024-08-20T22:08:11.1238221Z set +e 2024-08-20T22:08:11.1238527Z set -x 2024-08-20T22:08:11.1238862Z  2024-08-20T22:08:11.1239191Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:11.1239966Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:08:11.1241020Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:08:11.1241756Z # one is returned at random 2024-08-20T22:08:11.1242473Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:08:11.1242981Z  2024-08-20T22:08:11.1243308Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:11.1243928Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:08:11.1244680Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:08:11.1245224Z  2024-08-20T22:08:11.1245554Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:11.1246108Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:08:11.1246613Z  exit 1 2024-08-20T22:08:11.1246925Z  fi 2024-08-20T22:08:11.1247214Z fi 2024-08-20T22:08:11.1247500Z  2024-08-20T22:08:11.1247863Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:08:11.1248373Z  # Clean up any old records first 2024-08-20T22:08:11.1248891Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:08:11.1249393Z fi 2024-08-20T22:08:11.1249667Z  2024-08-20T22:08:11.1250103Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:08:11.1250648Z cat /etc/hosts 2024-08-20T22:08:11.1259528Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:11.1260038Z env: 2024-08-20T22:08:11.1260316Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:11.1260666Z ##[endgroup] 2024-08-20T22:08:11.1289562Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:11.1296029Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:08:11.1296507Z ++ tail -n1 2024-08-20T22:08:11.1693705Z + RESOLVED_IP=18.160.10.28 2024-08-20T22:08:11.1694248Z + '[' -z 18.160.10.28 ']' 2024-08-20T22:08:11.1694692Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:08:11.1711703Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T22:08:11.1712553Z + sudo tee -a /etc/hosts 2024-08-20T22:08:11.3198084Z 18.160.10.28 download.pytorch.org 2024-08-20T22:08:11.3220018Z + cat /etc/hosts 2024-08-20T22:08:11.3231199Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:08:11.3243062Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:08:11.3243768Z 18.160.10.28 download.pytorch.org 2024-08-20T22:08:11.3396879Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:08:11.3397476Z with: 2024-08-20T22:08:11.3398568Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3399832Z docker-build-dir: .ci/docker 2024-08-20T22:08:11.3400211Z working-directory: . 2024-08-20T22:08:11.3400682Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:11.3401203Z force-push: false 2024-08-20T22:08:11.3401509Z env: 2024-08-20T22:08:11.3401783Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:11.3402121Z ##[endgroup] 2024-08-20T22:08:11.3420664Z ##[group]Run set -ex 2024-08-20T22:08:11.3421022Z set -ex 2024-08-20T22:08:11.3421312Z  2024-08-20T22:08:11.3422048Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:08:11.3423007Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:08:11.3423798Z # job could then download the pre-built image as usual 2024-08-20T22:08:11.3424529Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:08:11.3425193Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3425821Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3426373Z  2024-08-20T22:08:11.3426878Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:08:11.3427496Z  exit 0 2024-08-20T22:08:11.3427875Z else 2024-08-20T22:08:11.3428240Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3428688Z fi 2024-08-20T22:08:11.3428978Z  2024-08-20T22:08:11.3429481Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:08:11.3430332Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:08:11.3431103Z  # use it as it is, but first let's extract the tag 2024-08-20T22:08:11.3431805Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:08:11.3432524Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3433216Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3433764Z else 2024-08-20T22:08:11.3434243Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:08:11.3435141Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3436142Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3436942Z fi 2024-08-20T22:08:11.3447311Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:11.3447970Z env: 2024-08-20T22:08:11.3448334Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:11.3448800Z REPO_NAME: pytorch 2024-08-20T22:08:11.3450070Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3451364Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:11.3451950Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:11.3452602Z ##[endgroup] 2024-08-20T22:08:11.3485681Z + [[ ! -d .ci/docker ]] 2024-08-20T22:08:11.3486285Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:08:11.3486754Z + echo skip=false 2024-08-20T22:08:11.3488822Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:08:11.3492301Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3493648Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:08:11.3519158Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3520000Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3521561Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3557662Z ##[group]Run set +e 2024-08-20T22:08:11.3558150Z set +e 2024-08-20T22:08:11.3558619Z set -x 2024-08-20T22:08:11.3558988Z  2024-08-20T22:08:11.3559405Z login() { 2024-08-20T22:08:11.3560455Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:11.3561273Z } 2024-08-20T22:08:11.3561635Z  2024-08-20T22:08:11.3562066Z retry () { 2024-08-20T22:08:11.3562635Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:11.3563244Z } 2024-08-20T22:08:11.3563582Z  2024-08-20T22:08:11.3564031Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:11.3564585Z  2024-08-20T22:08:11.3565126Z # Check if image already exists, if it does then skip building it 2024-08-20T22:08:11.3565942Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:08:11.3566601Z  exit 0 2024-08-20T22:08:11.3567006Z fi 2024-08-20T22:08:11.3567367Z  2024-08-20T22:08:11.3568012Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:08:11.3568941Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:08:11.3569881Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:08:11.3570676Z  # if we're on the base branch then use the parent commit 2024-08-20T22:08:11.3571382Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:08:11.3571908Z else 2024-08-20T22:08:11.3572530Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:08:11.3573337Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:08:11.3573899Z fi 2024-08-20T22:08:11.3574337Z  2024-08-20T22:08:11.3574779Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:08:11.3575343Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3575958Z  2024-08-20T22:08:11.3576808Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:08:11.3577660Z  exit 0 2024-08-20T22:08:11.3578103Z fi 2024-08-20T22:08:11.3578477Z  2024-08-20T22:08:11.3579062Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:08:11.3580210Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:08:11.3581066Z  exit 1 2024-08-20T22:08:11.3581461Z fi 2024-08-20T22:08:11.3581905Z  2024-08-20T22:08:11.3582491Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:08:11.3583493Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:08:11.3584532Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:08:11.3585648Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:08:11.3586902Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:08:11.3587935Z fi 2024-08-20T22:08:11.3588286Z  2024-08-20T22:08:11.3588821Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:11.3598037Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:11.3598708Z env: 2024-08-20T22:08:11.3599055Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:11.3599548Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:11.3600243Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:08:11.3601620Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3602931Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.3603691Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:11.3604430Z ##[endgroup] 2024-08-20T22:08:11.3634883Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:11.3635619Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:11.3638070Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:11.3639582Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:11.9101049Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:11.9102071Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:11.9103137Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:11.9103764Z 2024-08-20T22:08:11.9104466Z Login Succeeded 2024-08-20T22:08:11.9129755Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:12.1308759Z { 2024-08-20T22:08:12.1309501Z "schemaVersion": 2, 2024-08-20T22:08:12.1310381Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:08:12.1311328Z "config": { 2024-08-20T22:08:12.1311983Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:08:12.1312822Z "size": 48711, 2024-08-20T22:08:12.1313723Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:08:12.1314658Z }, 2024-08-20T22:08:12.1315100Z "layers": [ 2024-08-20T22:08:12.1315628Z { 2024-08-20T22:08:12.1316320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1317228Z "size": 28580681, 2024-08-20T22:08:12.1318088Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:08:12.1319319Z }, 2024-08-20T22:08:12.1319832Z { 2024-08-20T22:08:12.1320505Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1321339Z "size": 7943451, 2024-08-20T22:08:12.1322178Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:08:12.1323106Z }, 2024-08-20T22:08:12.1323535Z { 2024-08-20T22:08:12.1324219Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1325065Z "size": 55728572, 2024-08-20T22:08:12.1325879Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:08:12.1326848Z }, 2024-08-20T22:08:12.1327260Z { 2024-08-20T22:08:12.1327906Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1328826Z "size": 186, 2024-08-20T22:08:12.1329673Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:08:12.1330561Z }, 2024-08-20T22:08:12.1331090Z { 2024-08-20T22:08:12.1331771Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1332561Z "size": 6886, 2024-08-20T22:08:12.1333468Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:08:12.1334398Z }, 2024-08-20T22:08:12.1335249Z { 2024-08-20T22:08:12.1335966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1336813Z "size": 1329001756, 2024-08-20T22:08:12.1337958Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:08:12.1338823Z }, 2024-08-20T22:08:12.1339126Z { 2024-08-20T22:08:12.1339663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1340343Z "size": 62501, 2024-08-20T22:08:12.1340970Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:08:12.1341648Z }, 2024-08-20T22:08:12.1342035Z { 2024-08-20T22:08:12.1342570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1343162Z "size": 1684, 2024-08-20T22:08:12.1344151Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:08:12.1345144Z }, 2024-08-20T22:08:12.1345479Z { 2024-08-20T22:08:12.1346272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1346934Z "size": 1523, 2024-08-20T22:08:12.1347570Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:08:12.1348400Z }, 2024-08-20T22:08:12.1348757Z { 2024-08-20T22:08:12.1349289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1349945Z "size": 2528295403, 2024-08-20T22:08:12.1350648Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:08:12.1351306Z }, 2024-08-20T22:08:12.1351659Z { 2024-08-20T22:08:12.1352237Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1352885Z "size": 86016, 2024-08-20T22:08:12.1353483Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:08:12.1354242Z }, 2024-08-20T22:08:12.1354598Z { 2024-08-20T22:08:12.1355083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1355774Z "size": 1822, 2024-08-20T22:08:12.1356424Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:08:12.1357165Z }, 2024-08-20T22:08:12.1357468Z { 2024-08-20T22:08:12.1358005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1358687Z "size": 246761563, 2024-08-20T22:08:12.1359364Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:08:12.1360039Z }, 2024-08-20T22:08:12.1360421Z { 2024-08-20T22:08:12.1360953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1361583Z "size": 544, 2024-08-20T22:08:12.1362196Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:08:12.1362909Z }, 2024-08-20T22:08:12.1363243Z { 2024-08-20T22:08:12.1363771Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1364429Z "size": 1258, 2024-08-20T22:08:12.1365043Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:08:12.1365795Z }, 2024-08-20T22:08:12.1366094Z { 2024-08-20T22:08:12.1366607Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1367309Z "size": 484, 2024-08-20T22:08:12.1367910Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:08:12.1368553Z }, 2024-08-20T22:08:12.1368958Z { 2024-08-20T22:08:12.1369471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1370067Z "size": 91711766, 2024-08-20T22:08:12.1370771Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:08:12.1371458Z }, 2024-08-20T22:08:12.1371813Z { 2024-08-20T22:08:12.1372354Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1372982Z "size": 3390, 2024-08-20T22:08:12.1373772Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:08:12.1374520Z }, 2024-08-20T22:08:12.1374819Z { 2024-08-20T22:08:12.1375357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1376028Z "size": 1909, 2024-08-20T22:08:12.1376663Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:08:12.1377312Z }, 2024-08-20T22:08:12.1377691Z { 2024-08-20T22:08:12.1378228Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1378830Z "size": 701, 2024-08-20T22:08:12.1379550Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:12.1380274Z }, 2024-08-20T22:08:12.1380657Z { 2024-08-20T22:08:12.1381134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1381898Z "size": 2864254879, 2024-08-20T22:08:12.1382606Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:08:12.1383332Z }, 2024-08-20T22:08:12.1383633Z { 2024-08-20T22:08:12.1384195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1384849Z "size": 380, 2024-08-20T22:08:12.1385464Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:08:12.1386171Z }, 2024-08-20T22:08:12.1386524Z { 2024-08-20T22:08:12.1387037Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1387778Z "size": 12900, 2024-08-20T22:08:12.1388427Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:08:12.1389133Z }, 2024-08-20T22:08:12.1389557Z { 2024-08-20T22:08:12.1390073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1390733Z "size": 803, 2024-08-20T22:08:12.1391371Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:08:12.1392053Z }, 2024-08-20T22:08:12.1392407Z { 2024-08-20T22:08:12.1392976Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1393573Z "size": 106, 2024-08-20T22:08:12.1394207Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:08:12.1394949Z }, 2024-08-20T22:08:12.1395250Z { 2024-08-20T22:08:12.1395786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1396468Z "size": 503, 2024-08-20T22:08:12.1397102Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:08:12.1397754Z }, 2024-08-20T22:08:12.1398137Z { 2024-08-20T22:08:12.1398669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1399352Z "size": 121477267, 2024-08-20T22:08:12.1399951Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:08:12.1400655Z }, 2024-08-20T22:08:12.1401033Z { 2024-08-20T22:08:12.1401513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1402163Z "size": 109, 2024-08-20T22:08:12.1402829Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:08:12.1403547Z }, 2024-08-20T22:08:12.1403844Z { 2024-08-20T22:08:12.1404405Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1405052Z "size": 490, 2024-08-20T22:08:12.1405659Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:08:12.1406422Z }, 2024-08-20T22:08:12.1406775Z { 2024-08-20T22:08:12.1407288Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1407979Z "size": 297, 2024-08-20T22:08:12.1408561Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:08:12.1409244Z }, 2024-08-20T22:08:12.1409758Z { 2024-08-20T22:08:12.1410240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1410866Z "size": 103, 2024-08-20T22:08:12.1411555Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:08:12.1412249Z }, 2024-08-20T22:08:12.1412567Z { 2024-08-20T22:08:12.1413126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1413754Z "size": 1473, 2024-08-20T22:08:12.1414395Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:08:12.1415096Z }, 2024-08-20T22:08:12.1415426Z { 2024-08-20T22:08:12.1415960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1416608Z "size": 594729516, 2024-08-20T22:08:12.1417240Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:08:12.1418031Z }, 2024-08-20T22:08:12.1418415Z { 2024-08-20T22:08:12.1418921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1419547Z "size": 164, 2024-08-20T22:08:12.1420212Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:08:12.1420921Z }, 2024-08-20T22:08:12.1421218Z { 2024-08-20T22:08:12.1421824Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1422451Z "size": 564, 2024-08-20T22:08:12.1423109Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:08:12.1423779Z }, 2024-08-20T22:08:12.1424107Z { 2024-08-20T22:08:12.1424665Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1425282Z "size": 35874203, 2024-08-20T22:08:12.1425928Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:08:12.1426675Z }, 2024-08-20T22:08:12.1427026Z { 2024-08-20T22:08:12.1427510Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1428303Z "size": 104, 2024-08-20T22:08:12.1428913Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:08:12.1429598Z }, 2024-08-20T22:08:12.1429968Z { 2024-08-20T22:08:12.1430478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1431100Z "size": 425, 2024-08-20T22:08:12.1431801Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:08:12.1432473Z }, 2024-08-20T22:08:12.1432804Z { 2024-08-20T22:08:12.1433375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1433981Z "size": 20262183, 2024-08-20T22:08:12.1434638Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:08:12.1435382Z }, 2024-08-20T22:08:12.1435715Z { 2024-08-20T22:08:12.1436214Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1436900Z "size": 438, 2024-08-20T22:08:12.1437855Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:08:12.1438576Z }, 2024-08-20T22:08:12.1438956Z { 2024-08-20T22:08:12.1439471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1440122Z "size": 701, 2024-08-20T22:08:12.1440753Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:12.1441464Z }, 2024-08-20T22:08:12.1441795Z { 2024-08-20T22:08:12.1442351Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1442964Z "size": 141, 2024-08-20T22:08:12.1443592Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:08:12.1444339Z }, 2024-08-20T22:08:12.1444694Z { 2024-08-20T22:08:12.1445178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1445852Z "size": 135, 2024-08-20T22:08:12.1446639Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:08:12.1447379Z }, 2024-08-20T22:08:12.1447698Z { 2024-08-20T22:08:12.1448208Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1448888Z "size": 4838068375, 2024-08-20T22:08:12.1449512Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:08:12.1450199Z }, 2024-08-20T22:08:12.1450594Z { 2024-08-20T22:08:12.1451107Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1451697Z "size": 194, 2024-08-20T22:08:12.1452378Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:08:12.1453063Z }, 2024-08-20T22:08:12.1453393Z { 2024-08-20T22:08:12.1453931Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1454690Z "size": 564, 2024-08-20T22:08:12.1455342Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:08:12.1456052Z }, 2024-08-20T22:08:12.1456384Z { 2024-08-20T22:08:12.1456916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1457596Z "size": 43162921, 2024-08-20T22:08:12.1458191Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:08:12.1458895Z }, 2024-08-20T22:08:12.1459275Z { 2024-08-20T22:08:12.1459809Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1460405Z "size": 106, 2024-08-20T22:08:12.1461061Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:08:12.1461767Z }, 2024-08-20T22:08:12.1462098Z { 2024-08-20T22:08:12.1462636Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1463292Z "size": 1380, 2024-08-20T22:08:12.1463906Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:08:12.1464617Z }, 2024-08-20T22:08:12.1464970Z { 2024-08-20T22:08:12.1465480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1466171Z "size": 701, 2024-08-20T22:08:12.1466755Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:12.1467451Z }, 2024-08-20T22:08:12.1467913Z { 2024-08-20T22:08:12.1468429Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1469023Z "size": 139, 2024-08-20T22:08:12.1469758Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:08:12.1470445Z }, 2024-08-20T22:08:12.1470742Z { 2024-08-20T22:08:12.1471320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1471949Z "size": 118, 2024-08-20T22:08:12.1472634Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:08:12.1473297Z }, 2024-08-20T22:08:12.1473629Z { 2024-08-20T22:08:12.1474205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1474838Z "size": 4324672728, 2024-08-20T22:08:12.1475458Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:08:12.1476225Z }, 2024-08-20T22:08:12.1476557Z { 2024-08-20T22:08:12.1477035Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1477726Z "size": 174, 2024-08-20T22:08:12.1478327Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:08:12.1479048Z }, 2024-08-20T22:08:12.1479418Z { 2024-08-20T22:08:12.1479929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1480574Z "size": 907, 2024-08-20T22:08:12.1481264Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:08:12.1481929Z }, 2024-08-20T22:08:12.1482380Z { 2024-08-20T22:08:12.1482953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1483602Z "size": 701, 2024-08-20T22:08:12.1484184Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:12.1484935Z }, 2024-08-20T22:08:12.1485285Z { 2024-08-20T22:08:12.1485766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1486452Z "size": 134, 2024-08-20T22:08:12.1487096Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:08:12.1487834Z }, 2024-08-20T22:08:12.1488185Z { 2024-08-20T22:08:12.1488730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1489371Z "size": 32, 2024-08-20T22:08:12.1490069Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:12.1490836Z }, 2024-08-20T22:08:12.1491179Z { 2024-08-20T22:08:12.1491777Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1492420Z "size": 158, 2024-08-20T22:08:12.1493016Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:08:12.1493784Z }, 2024-08-20T22:08:12.1494155Z { 2024-08-20T22:08:12.1494660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1495346Z "size": 1841, 2024-08-20T22:08:12.1495981Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:08:12.1496742Z }, 2024-08-20T22:08:12.1497049Z { 2024-08-20T22:08:12.1497571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1498277Z "size": 7529776, 2024-08-20T22:08:12.1498917Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:08:12.1499616Z }, 2024-08-20T22:08:12.1500051Z { 2024-08-20T22:08:12.1500580Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1501232Z "size": 163, 2024-08-20T22:08:12.1501899Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:08:12.1502598Z }, 2024-08-20T22:08:12.1502962Z { 2024-08-20T22:08:12.1503510Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1504159Z "size": 7943, 2024-08-20T22:08:12.1504822Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:08:12.1505580Z }, 2024-08-20T22:08:12.1505892Z { 2024-08-20T22:08:12.1506430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1507115Z "size": 8068, 2024-08-20T22:08:12.1507885Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:08:12.1508554Z }, 2024-08-20T22:08:12.1508957Z { 2024-08-20T22:08:12.1509551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1510158Z "size": 302, 2024-08-20T22:08:12.1510847Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:08:12.1511555Z }, 2024-08-20T22:08:12.1511896Z { 2024-08-20T22:08:12.1512432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1513096Z "size": 7629638, 2024-08-20T22:08:12.1513728Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:08:12.1514477Z }, 2024-08-20T22:08:12.1514806Z { 2024-08-20T22:08:12.1515326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1516033Z "size": 108, 2024-08-20T22:08:12.1516632Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:08:12.1517336Z }, 2024-08-20T22:08:12.1517740Z { 2024-08-20T22:08:12.1518270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1518876Z "size": 54145661, 2024-08-20T22:08:12.1519756Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:08:12.1520468Z }, 2024-08-20T22:08:12.1520883Z { 2024-08-20T22:08:12.1521399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1522042Z "size": 473, 2024-08-20T22:08:12.1522746Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:08:12.1523448Z }, 2024-08-20T22:08:12.1523777Z { 2024-08-20T22:08:12.1524339Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1524980Z "size": 1374858766, 2024-08-20T22:08:12.1525617Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:08:12.1526360Z }, 2024-08-20T22:08:12.1526699Z { 2024-08-20T22:08:12.1527242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1528006Z "size": 106, 2024-08-20T22:08:12.1528676Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:08:12.1529414Z }, 2024-08-20T22:08:12.1529857Z { 2024-08-20T22:08:12.1530359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1530998Z "size": 559, 2024-08-20T22:08:12.1531465Z + exit 0 2024-08-20T22:08:12.1532082Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:08:12.1532732Z }, 2024-08-20T22:08:12.1533143Z { 2024-08-20T22:08:12.1533665Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1534329Z "size": 46248488, 2024-08-20T22:08:12.1535005Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:08:12.1535710Z }, 2024-08-20T22:08:12.1536054Z { 2024-08-20T22:08:12.1536612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1537493Z "size": 111, 2024-08-20T22:08:12.1538156Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:08:12.1723869Z }, 2024-08-20T22:08:12.1724292Z { 2024-08-20T22:08:12.1724750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1725309Z "size": 32, 2024-08-20T22:08:12.1725848Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:12.1726475Z }, 2024-08-20T22:08:12.1726723Z { 2024-08-20T22:08:12.1727154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1727695Z "size": 32, 2024-08-20T22:08:12.1728227Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:12.1728847Z }, 2024-08-20T22:08:12.1729118Z { 2024-08-20T22:08:12.1729567Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1730133Z "size": 32, 2024-08-20T22:08:12.1730672Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:12.1731290Z }, 2024-08-20T22:08:12.1731529Z { 2024-08-20T22:08:12.1731955Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:12.1732500Z "size": 32, 2024-08-20T22:08:12.1733024Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:12.1733636Z } 2024-08-20T22:08:12.1733881Z ] 2024-08-20T22:08:12.1734125Z } 2024-08-20T22:08:12.1835668Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:12.1836165Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:12.1836708Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:08:12.1846352Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:12.1846852Z env: 2024-08-20T22:08:12.1847135Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:12.1848285Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:12.1849506Z ##[endgroup] 2024-08-20T22:08:12.1881194Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:12.1934947Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:08:12.1935520Z with: 2024-08-20T22:08:12.1936575Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:12.1938237Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:12.1938757Z env: 2024-08-20T22:08:12.1939036Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:12.1939388Z ##[endgroup] 2024-08-20T22:08:12.1956816Z ##[group]Run set -x 2024-08-20T22:08:12.1957342Z set -x 2024-08-20T22:08:12.1957643Z set +e 2024-08-20T22:08:12.1957942Z  2024-08-20T22:08:12.1958214Z login() { 2024-08-20T22:08:12.1958863Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:12.1959568Z } 2024-08-20T22:08:12.1959848Z  2024-08-20T22:08:12.1960153Z retry () { 2024-08-20T22:08:12.1960544Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:12.1960985Z } 2024-08-20T22:08:12.1961261Z  2024-08-20T22:08:12.1961580Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:12.1961992Z  2024-08-20T22:08:12.1962266Z set -e 2024-08-20T22:08:12.1962737Z # ignore output since only exit code is used for conditional 2024-08-20T22:08:12.1963432Z # only pull docker image if it's not available locally 2024-08-20T22:08:12.1964183Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:08:12.1964881Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:08:12.1965321Z fi 2024-08-20T22:08:12.1974223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:12.1974721Z env: 2024-08-20T22:08:12.1975005Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:12.1976140Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:12.1977425Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:12.1977933Z ##[endgroup] 2024-08-20T22:08:12.2007574Z + set +e 2024-08-20T22:08:12.2008238Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:12.2008883Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:12.2011358Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:12.2012553Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:12.7464243Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:12.7465303Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:12.7466396Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:12.7467072Z 2024-08-20T22:08:12.7467236Z Login Succeeded 2024-08-20T22:08:12.7487632Z + set -e 2024-08-20T22:08:12.7488947Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:12.7652289Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:12.7654410Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:13.0083462Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:08:13.0084840Z 7a2c55901189: Pulling fs layer 2024-08-20T22:08:13.0085399Z 224fe954d725: Pulling fs layer 2024-08-20T22:08:13.0086477Z 75722010b82e: Pulling fs layer 2024-08-20T22:08:13.0087055Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:08:13.0087611Z b57676e46aee: Pulling fs layer 2024-08-20T22:08:13.0088145Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:08:13.0088660Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:08:13.0089201Z 0c1227890755: Pulling fs layer 2024-08-20T22:08:13.0089746Z d8d1234baab3: Pulling fs layer 2024-08-20T22:08:13.0090252Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:08:13.0090795Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:08:13.0091342Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:08:13.0091873Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:08:13.0092669Z 4373aa743caf: Pulling fs layer 2024-08-20T22:08:13.0093216Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:08:13.0093747Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:08:13.0094143Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:08:13.0094550Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:08:13.0095524Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:08:13.0096019Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:08:13.0096531Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:08:13.0097006Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:08:13.0097475Z 0c1227890755: Waiting 2024-08-20T22:08:13.0097884Z 7ed32bc8e469: Waiting 2024-08-20T22:08:13.0098288Z d8d1234baab3: Waiting 2024-08-20T22:08:13.0098727Z ec1e7978c1fe: Waiting 2024-08-20T22:08:13.0099182Z 33b639226eee: Pulling fs layer 2024-08-20T22:08:13.0099676Z d3547bd43146: Pulling fs layer 2024-08-20T22:08:13.0100135Z ef1e3ef1e85f: Waiting 2024-08-20T22:08:13.0100569Z 5645520256fa: Pulling fs layer 2024-08-20T22:08:13.0101038Z 2df849f72ba4: Waiting 2024-08-20T22:08:13.0101503Z a41a8d1c11c8: Waiting 2024-08-20T22:08:13.0101919Z 0bf61114d37b: Waiting 2024-08-20T22:08:13.0102297Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:08:13.0102655Z 4373aa743caf: Waiting 2024-08-20T22:08:13.0102988Z 385d0421d67c: Pulling fs layer 2024-08-20T22:08:13.0103339Z b57676e46aee: Waiting 2024-08-20T22:08:13.0103671Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:08:13.0104260Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:08:13.0104779Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:08:13.0105236Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:08:13.0105615Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:08:13.0105987Z 7754ee068b33: Pulling fs layer 2024-08-20T22:08:13.0106349Z d527cbbb87e3: Waiting 2024-08-20T22:08:13.0106680Z 0479fbac3361: Pulling fs layer 2024-08-20T22:08:13.0107065Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:08:13.0107445Z afc429e87b59: Pulling fs layer 2024-08-20T22:08:13.0107909Z 34c47d63c5de: Waiting 2024-08-20T22:08:13.0108233Z 6efb96fc87ea: Waiting 2024-08-20T22:08:13.0108576Z 019620471c25: Pulling fs layer 2024-08-20T22:08:13.0108934Z ac23c5fcc1ef: Waiting 2024-08-20T22:08:13.0109270Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:08:13.0109646Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:08:13.0109999Z d3547bd43146: Waiting 2024-08-20T22:08:13.0110306Z 5645520256fa: Waiting 2024-08-20T22:08:13.0110620Z 1fef4bd19792: Waiting 2024-08-20T22:08:13.0110937Z b119c4fc6c58: Waiting 2024-08-20T22:08:13.0111256Z 385d0421d67c: Waiting 2024-08-20T22:08:13.0111565Z 6658e558ebe4: Waiting 2024-08-20T22:08:13.0111998Z 7cbfce999d76: Waiting 2024-08-20T22:08:13.0112317Z 1811ff4d3556: Waiting 2024-08-20T22:08:13.0112631Z f1d6874dee9c: Waiting 2024-08-20T22:08:13.0112956Z 560cd6122a38: Pulling fs layer 2024-08-20T22:08:13.0113314Z 829ed7d91e38: Waiting 2024-08-20T22:08:13.0113749Z 7c47122f4e09: Waiting 2024-08-20T22:08:13.0114056Z 33444a5a7e01: Waiting 2024-08-20T22:08:13.0114384Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:08:13.0114744Z 27aeb1b2cd5e: Waiting 2024-08-20T22:08:13.0115165Z 829c0e19f4b7: Waiting 2024-08-20T22:08:13.0115710Z da6a838a66c7: Pulling fs layer 2024-08-20T22:08:13.0116084Z 5a4eb19d7219: Waiting 2024-08-20T22:08:13.0116415Z cbb79a705353: Pulling fs layer 2024-08-20T22:08:13.0116786Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:08:13.0117167Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:08:13.0117526Z 75b2abb16c68: Waiting 2024-08-20T22:08:13.0118072Z 4a0412373305: Pulling fs layer 2024-08-20T22:08:13.0118439Z 302c8c177944: Pulling fs layer 2024-08-20T22:08:13.0118809Z 02970347dff6: Pulling fs layer 2024-08-20T22:08:13.0119180Z da6a838a66c7: Waiting 2024-08-20T22:08:13.0119549Z 916c5e858016: Pulling fs layer 2024-08-20T22:08:13.0119915Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:08:13.0120294Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:08:13.0120648Z 302c8c177944: Waiting 2024-08-20T22:08:13.0120969Z f957c1624481: Pulling fs layer 2024-08-20T22:08:13.0121339Z 768c31dca016: Pulling fs layer 2024-08-20T22:08:13.0121692Z 02970347dff6: Waiting 2024-08-20T22:08:13.0122209Z 4d4fa6802ab3: Waiting 2024-08-20T22:08:13.0122524Z 916c5e858016: Waiting 2024-08-20T22:08:13.0122919Z 019620471c25: Waiting 2024-08-20T22:08:13.0123249Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:08:13.0123629Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:08:13.0123978Z f957c1624481: Waiting 2024-08-20T22:08:13.0124303Z 0f085d4fa279: Waiting 2024-08-20T22:08:13.0124648Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:08:13.0125009Z c42b3602a1fb: Waiting 2024-08-20T22:08:13.0125324Z 4a0412373305: Waiting 2024-08-20T22:08:13.0125634Z 768c31dca016: Waiting 2024-08-20T22:08:13.0125958Z 560cd6122a38: Waiting 2024-08-20T22:08:13.0126268Z 8c32d0ecebbe: Waiting 2024-08-20T22:08:13.0126594Z 33b639226eee: Waiting 2024-08-20T22:08:13.0126922Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:08:13.0127291Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:08:13.0127668Z b10de02255c0: Pulling fs layer 2024-08-20T22:08:13.0128026Z 37513e9b43d7: Waiting 2024-08-20T22:08:13.0128340Z a8c1e85b5e14: Waiting 2024-08-20T22:08:13.0128675Z e56eeff2db93: Pulling fs layer 2024-08-20T22:08:13.0129033Z 7754ee068b33: Waiting 2024-08-20T22:08:13.0129368Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:08:13.0129780Z b10de02255c0: Waiting 2024-08-20T22:08:13.0130111Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:08:13.0130488Z ee1273153675: Pulling fs layer 2024-08-20T22:08:13.0130851Z 0479fbac3361: Waiting 2024-08-20T22:08:13.0131184Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:08:13.0131543Z eb09cbf3e416: Waiting 2024-08-20T22:08:13.0131871Z ee1273153675: Waiting 2024-08-20T22:08:13.0132208Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:08:13.0132582Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:08:13.0132960Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:08:13.0133338Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:08:13.0133717Z d79fd7d52698: Pulling fs layer 2024-08-20T22:08:13.0134084Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:08:13.0134443Z e22979f9e0ee: Waiting 2024-08-20T22:08:13.0134862Z 5d115b29b5d5: Waiting 2024-08-20T22:08:13.0135193Z 7f2d7a04fbe6: Waiting 2024-08-20T22:08:13.0135508Z d79fd7d52698: Waiting 2024-08-20T22:08:13.0135835Z 7ebe49c3a23e: Waiting 2024-08-20T22:08:13.0136174Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:08:13.0136528Z 78ee77eddef2: Waiting 2024-08-20T22:08:13.0136844Z 4f4fb700ef54: Waiting 2024-08-20T22:08:13.0137511Z 6c1f0aabce7c: Waiting 2024-08-20T22:08:13.1534689Z 224fe954d725: Verifying Checksum 2024-08-20T22:08:13.1535276Z 224fe954d725: Download complete 2024-08-20T22:08:13.2349056Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:08:13.2349480Z d527cbbb87e3: Download complete 2024-08-20T22:08:13.3014098Z b57676e46aee: Verifying Checksum 2024-08-20T22:08:13.3014531Z b57676e46aee: Download complete 2024-08-20T22:08:13.3462795Z 7a2c55901189: Verifying Checksum 2024-08-20T22:08:13.3463211Z 7a2c55901189: Download complete 2024-08-20T22:08:13.4138515Z a41a8d1c11c8: Download complete 2024-08-20T22:08:13.4965504Z 0c1227890755: Verifying Checksum 2024-08-20T22:08:13.4966091Z 0c1227890755: Download complete 2024-08-20T22:08:13.5766861Z d8d1234baab3: Download complete 2024-08-20T22:08:13.6224168Z 75722010b82e: Verifying Checksum 2024-08-20T22:08:13.6224746Z 75722010b82e: Download complete 2024-08-20T22:08:13.7123610Z ec1e7978c1fe: Download complete 2024-08-20T22:08:13.7906895Z ac23c5fcc1ef: Download complete 2024-08-20T22:08:14.5328994Z 7a2c55901189: Pull complete 2024-08-20T22:08:14.8498315Z 224fe954d725: Pull complete 2024-08-20T22:08:15.7603905Z 75722010b82e: Pull complete 2024-08-20T22:08:15.7832130Z d527cbbb87e3: Pull complete 2024-08-20T22:08:15.8046848Z b57676e46aee: Pull complete 2024-08-20T22:08:16.3050979Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:08:16.3051620Z ca426ffc9d2b: Download complete 2024-08-20T22:08:16.3789999Z 4373aa743caf: Download complete 2024-08-20T22:08:16.4508470Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:08:16.4509046Z 75b2abb16c68: Download complete 2024-08-20T22:08:16.5184032Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:08:16.5184477Z ef1e3ef1e85f: Download complete 2024-08-20T22:08:17.5134317Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:08:17.5134794Z 0bf61114d37b: Download complete 2024-08-20T22:08:17.6144329Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:08:17.6144763Z 2df849f72ba4: Download complete 2024-08-20T22:08:17.7027064Z 34c47d63c5de: Download complete 2024-08-20T22:08:17.7761063Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:08:17.7761559Z 1fef4bd19792: Download complete 2024-08-20T22:08:26.6524431Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:08:26.6525038Z a8c1e85b5e14: Download complete 2024-08-20T22:08:26.7309855Z b119c4fc6c58: Download complete 2024-08-20T22:08:26.8088528Z 33b639226eee: Verifying Checksum 2024-08-20T22:08:26.8089156Z 33b639226eee: Download complete 2024-08-20T22:08:26.8868683Z d3547bd43146: Verifying Checksum 2024-08-20T22:08:26.8869318Z d3547bd43146: Download complete 2024-08-20T22:08:26.9778835Z 5645520256fa: Verifying Checksum 2024-08-20T22:08:26.9779433Z 5645520256fa: Download complete 2024-08-20T22:08:27.0639335Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:08:27.0639966Z 829ed7d91e38: Download complete 2024-08-20T22:08:28.3191173Z 385d0421d67c: Verifying Checksum 2024-08-20T22:08:28.3191765Z 385d0421d67c: Download complete 2024-08-20T22:08:28.3956780Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:08:28.3957260Z 7cbfce999d76: Download complete 2024-08-20T22:08:28.4698113Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:08:28.4698725Z 6658e558ebe4: Download complete 2024-08-20T22:08:28.5426488Z 33444a5a7e01: Download complete 2024-08-20T22:08:28.6094717Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:08:28.6095316Z 7c47122f4e09: Download complete 2024-08-20T22:08:28.6802418Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:08:28.6802990Z 1811ff4d3556: Download complete 2024-08-20T22:08:34.6783887Z 7754ee068b33: Verifying Checksum 2024-08-20T22:08:34.6784447Z 7754ee068b33: Download complete 2024-08-20T22:08:34.7395069Z 0479fbac3361: Verifying Checksum 2024-08-20T22:08:34.7395505Z 0479fbac3361: Download complete 2024-08-20T22:08:34.8541581Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:08:34.8542003Z f1d6874dee9c: Download complete 2024-08-20T22:08:35.2711766Z afc429e87b59: Verifying Checksum 2024-08-20T22:08:35.2712249Z afc429e87b59: Download complete 2024-08-20T22:08:35.3376655Z 019620471c25: Download complete 2024-08-20T22:08:35.4135006Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:08:35.4135622Z 27aeb1b2cd5e: Download complete 2024-08-20T22:08:35.7014521Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:08:35.7015501Z 829c0e19f4b7: Download complete 2024-08-20T22:08:35.7711125Z 560cd6122a38: Verifying Checksum 2024-08-20T22:08:35.7711552Z 560cd6122a38: Download complete 2024-08-20T22:08:35.8375088Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:08:35.8375747Z 5a4eb19d7219: Download complete 2024-08-20T22:08:35.9432719Z da6a838a66c7: Verifying Checksum 2024-08-20T22:08:35.9433231Z da6a838a66c7: Download complete 2024-08-20T22:08:38.9087131Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:08:38.9087806Z 7ed32bc8e469: Download complete 2024-08-20T22:08:38.9826843Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:08:38.9827352Z 4d4fa6802ab3: Download complete 2024-08-20T22:08:39.0506527Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:08:39.0506960Z 0f085d4fa279: Download complete 2024-08-20T22:08:39.5371992Z 4a0412373305: Verifying Checksum 2024-08-20T22:08:39.5372582Z 4a0412373305: Download complete 2024-08-20T22:08:39.6219807Z 302c8c177944: Download complete 2024-08-20T22:08:39.7140892Z 02970347dff6: Verifying Checksum 2024-08-20T22:08:39.7141506Z 02970347dff6: Download complete 2024-08-20T22:08:39.7873524Z 916c5e858016: Verifying Checksum 2024-08-20T22:08:39.7874159Z 916c5e858016: Download complete 2024-08-20T22:08:39.8630750Z c42b3602a1fb: Download complete 2024-08-20T22:08:39.9802609Z a8c1e85b5e14: Pull complete 2024-08-20T22:08:40.2170355Z a41a8d1c11c8: Pull complete 2024-08-20T22:08:40.4486124Z 0c1227890755: Pull complete 2024-08-20T22:08:40.6839744Z d8d1234baab3: Pull complete 2024-08-20T22:08:46.4598495Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:08:46.4599448Z 6efb96fc87ea: Download complete 2024-08-20T22:08:46.5477667Z f957c1624481: Verifying Checksum 2024-08-20T22:08:46.5478275Z f957c1624481: Download complete 2024-08-20T22:08:46.6234968Z 768c31dca016: Verifying Checksum 2024-08-20T22:08:46.6235436Z 768c31dca016: Download complete 2024-08-20T22:08:46.7017418Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:08:46.7018116Z 8c32d0ecebbe: Download complete 2024-08-20T22:08:46.7116002Z 4f4fb700ef54: Download complete 2024-08-20T22:08:46.8044603Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:08:46.8045064Z 78ee77eddef2: Download complete 2024-08-20T22:08:46.8900732Z 37513e9b43d7: Download complete 2024-08-20T22:08:47.0275079Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:08:47.0275558Z a7a5392e2bf6: Download complete 2024-08-20T22:08:47.1013368Z b10de02255c0: Download complete 2024-08-20T22:08:47.1665524Z e56eeff2db93: Verifying Checksum 2024-08-20T22:08:47.1666152Z e56eeff2db93: Download complete 2024-08-20T22:08:47.2404207Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:08:47.2404691Z 70ac3e38d0fd: Download complete 2024-08-20T22:08:47.3234010Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:08:47.3234455Z eb09cbf3e416: Download complete 2024-08-20T22:08:47.4592035Z ee1273153675: Verifying Checksum 2024-08-20T22:08:47.4592636Z ee1273153675: Download complete 2024-08-20T22:08:47.5451009Z e22979f9e0ee: Download complete 2024-08-20T22:08:48.1459365Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:08:48.1459832Z 6c1f0aabce7c: Download complete 2024-08-20T22:08:48.2594517Z 7ebe49c3a23e: Download complete 2024-08-20T22:09:16.1893194Z 5d115b29b5d5: Download complete 2024-08-20T22:09:16.2553385Z 7f2d7a04fbe6: Download complete 2024-08-20T22:09:16.3361885Z d79fd7d52698: Verifying Checksum 2024-08-20T22:09:16.3362316Z d79fd7d52698: Download complete 2024-08-20T22:09:17.8288495Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:09:17.8288963Z 9752d6fbbcab: Download complete 2024-08-20T22:09:17.9086106Z 83808b2eff4d: Download complete 2024-08-20T22:09:56.2579773Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:09:56.2580402Z 11fad972ef6c: Download complete 2024-08-20T22:09:56.6180889Z cbb79a705353: Download complete 2024-08-20T22:10:12.1754895Z 7ed32bc8e469: Pull complete 2024-08-20T22:10:12.4044456Z ec1e7978c1fe: Pull complete 2024-08-20T22:10:12.6375822Z ac23c5fcc1ef: Pull complete 2024-08-20T22:10:21.2464392Z ca426ffc9d2b: Pull complete 2024-08-20T22:10:21.4816199Z 4373aa743caf: Pull complete 2024-08-20T22:10:21.7275869Z 75b2abb16c68: Pull complete 2024-08-20T22:10:21.9544140Z ef1e3ef1e85f: Pull complete 2024-08-20T22:10:24.6337841Z 0bf61114d37b: Pull complete 2024-08-20T22:10:24.8542858Z 2df849f72ba4: Pull complete 2024-08-20T22:10:25.0851754Z 34c47d63c5de: Pull complete 2024-08-20T22:10:25.3103183Z 1fef4bd19792: Pull complete 2024-08-20T22:11:23.6883941Z 6efb96fc87ea: Pull complete 2024-08-20T22:11:23.8509736Z b119c4fc6c58: Pull complete 2024-08-20T22:11:24.0260628Z 33b639226eee: Pull complete 2024-08-20T22:11:24.1905283Z d3547bd43146: Pull complete 2024-08-20T22:11:24.3885514Z 5645520256fa: Pull complete 2024-08-20T22:11:24.6180404Z 829ed7d91e38: Pull complete 2024-08-20T22:11:28.1307809Z 385d0421d67c: Pull complete 2024-08-20T22:11:28.2579849Z 7cbfce999d76: Pull complete 2024-08-20T22:11:28.4272019Z 6658e558ebe4: Pull complete 2024-08-20T22:11:28.6673553Z 33444a5a7e01: Pull complete 2024-08-20T22:11:28.8164141Z 7c47122f4e09: Pull complete 2024-08-20T22:11:29.0125113Z 1811ff4d3556: Pull complete 2024-08-20T22:11:39.7618061Z 7754ee068b33: Pull complete 2024-08-20T22:11:39.9738477Z 0479fbac3361: Pull complete 2024-08-20T22:11:40.1687222Z f1d6874dee9c: Pull complete 2024-08-20T22:11:41.1221140Z afc429e87b59: Pull complete 2024-08-20T22:11:41.3258415Z 019620471c25: Pull complete 2024-08-20T22:11:41.5234042Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:11:41.9984571Z 829c0e19f4b7: Pull complete 2024-08-20T22:11:42.1940124Z 560cd6122a38: Pull complete 2024-08-20T22:11:42.5681829Z 5a4eb19d7219: Pull complete 2024-08-20T22:11:42.6755824Z da6a838a66c7: Pull complete 2024-08-20T22:12:21.5912562Z cbb79a705353: Pull complete 2024-08-20T22:12:21.8163264Z 4d4fa6802ab3: Pull complete 2024-08-20T22:12:22.0441697Z 0f085d4fa279: Pull complete 2024-08-20T22:12:24.4135864Z 4a0412373305: Pull complete 2024-08-20T22:12:24.4276455Z 302c8c177944: Pull complete 2024-08-20T22:12:24.4417112Z 02970347dff6: Pull complete 2024-08-20T22:12:24.4731268Z 916c5e858016: Pull complete 2024-08-20T22:12:24.4880006Z c42b3602a1fb: Pull complete 2024-08-20T22:13:46.8943481Z 11fad972ef6c: Pull complete 2024-08-20T22:13:47.1007252Z f957c1624481: Pull complete 2024-08-20T22:13:47.2988079Z 768c31dca016: Pull complete 2024-08-20T22:13:47.6722364Z 8c32d0ecebbe: Pull complete 2024-08-20T22:13:47.8631294Z 4f4fb700ef54: Pull complete 2024-08-20T22:13:47.9772153Z 78ee77eddef2: Pull complete 2024-08-20T22:13:48.1159784Z 37513e9b43d7: Pull complete 2024-08-20T22:13:48.5500599Z a7a5392e2bf6: Pull complete 2024-08-20T22:13:48.6050840Z b10de02255c0: Pull complete 2024-08-20T22:13:48.7600028Z e56eeff2db93: Pull complete 2024-08-20T22:13:48.9507905Z 70ac3e38d0fd: Pull complete 2024-08-20T22:13:49.1429164Z eb09cbf3e416: Pull complete 2024-08-20T22:13:50.4832647Z ee1273153675: Pull complete 2024-08-20T22:13:50.6739215Z e22979f9e0ee: Pull complete 2024-08-20T22:13:53.4203049Z 6c1f0aabce7c: Pull complete 2024-08-20T22:13:53.6240282Z 7ebe49c3a23e: Pull complete 2024-08-20T22:14:41.3161372Z 5d115b29b5d5: Pull complete 2024-08-20T22:14:41.4536627Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:14:41.5578724Z d79fd7d52698: Pull complete 2024-08-20T22:14:43.3920773Z 9752d6fbbcab: Pull complete 2024-08-20T22:14:43.4864588Z 83808b2eff4d: Pull complete 2024-08-20T22:14:44.3026130Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:14:44.3466292Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:44.3748396Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:44.3843367Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:44.3844591Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:44.3858716Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:14:44.3859368Z env: 2024-08-20T22:14:44.3859725Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:44.3860183Z ##[endgroup] 2024-08-20T22:14:44.4073746Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:14:44.4074299Z with: 2024-08-20T22:14:44.4074580Z driver-version: 550.54.15 2024-08-20T22:14:44.4074918Z env: 2024-08-20T22:14:44.4075191Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:44.4075548Z ##[endgroup] 2024-08-20T22:14:44.4206012Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:14:44.4206580Z with: 2024-08-20T22:14:44.4206852Z timeout_minutes: 10 2024-08-20T22:14:44.4207168Z max_attempts: 3 2024-08-20T22:14:44.4239616Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:14:44.4272244Z retry_wait_seconds: 10 2024-08-20T22:14:44.4272620Z polling_interval_seconds: 1 2024-08-20T22:14:44.4272991Z warning_on_retry: true 2024-08-20T22:14:44.4273350Z continue_on_error: false 2024-08-20T22:14:44.4273689Z env: 2024-08-20T22:14:44.4273966Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:44.4274319Z DRIVER_VERSION: 550.54.15 2024-08-20T22:14:44.4274666Z ##[endgroup] 2024-08-20T22:14:44.8019123Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:14:44.8021394Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:14:44.8022124Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:14:46.1244569Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:14:46.1245622Z No packages marked for removal. 2024-08-20T22:14:46.1629815Z Dependencies resolved. 2024-08-20T22:14:46.1639885Z Nothing to do. 2024-08-20T22:14:46.1641936Z Complete! 2024-08-20T22:14:46.2417874Z + install_nvidia_driver_common 2024-08-20T22:14:46.2422170Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:14:46.2423194Z Before installing NVIDIA driver 2024-08-20T22:14:46.2423783Z + lspci 2024-08-20T22:14:46.2532179Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:46.2532966Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:46.2533922Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:46.2535055Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:46.2536027Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:46.2536884Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:46.2537893Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:46.2538715Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:46.2539303Z + lsmod 2024-08-20T22:14:46.2783549Z Module Size Used by 2024-08-20T22:14:46.2784086Z xt_nat 16384 0 2024-08-20T22:14:46.2784498Z nvidia_modeset 1351680 0 2024-08-20T22:14:46.2784927Z video 65536 1 nvidia_modeset 2024-08-20T22:14:46.2785473Z wmi 36864 1 video 2024-08-20T22:14:46.2786086Z nvidia_uvm 4706304 0 2024-08-20T22:14:46.2786553Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:14:46.2787055Z drm 602112 1 nvidia 2024-08-20T22:14:46.2787505Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:14:46.2788343Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:14:46.2789016Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:14:46.2789546Z veth 36864 0 2024-08-20T22:14:46.2789928Z xt_conntrack 16384 1 2024-08-20T22:14:46.2790304Z nft_chain_nat 16384 3 2024-08-20T22:14:46.2790675Z xt_MASQUERADE 20480 1 2024-08-20T22:14:46.2791202Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:46.2791898Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:46.2792677Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:46.2793346Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:46.2793814Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:46.2794352Z xfrm_user 57344 1 2024-08-20T22:14:46.2795154Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:46.2795674Z xt_addrtype 16384 2 2024-08-20T22:14:46.2796079Z nft_compat 20480 4 2024-08-20T22:14:46.2796570Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:46.2797184Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:46.2797836Z br_netfilter 36864 0 2024-08-20T22:14:46.2798375Z bridge 307200 1 br_netfilter 2024-08-20T22:14:46.2798950Z stp 16384 1 bridge 2024-08-20T22:14:46.2799518Z llc 16384 2 bridge,stp 2024-08-20T22:14:46.2800060Z overlay 167936 0 2024-08-20T22:14:46.2800547Z tls 114688 0 2024-08-20T22:14:46.2800999Z nls_ascii 16384 1 2024-08-20T22:14:46.2801364Z nls_cp437 20480 1 2024-08-20T22:14:46.2801733Z vfat 24576 1 2024-08-20T22:14:46.2802104Z fat 86016 1 vfat 2024-08-20T22:14:46.2802509Z sunrpc 692224 1 2024-08-20T22:14:46.2802870Z ena 167936 0 2024-08-20T22:14:46.2803268Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:46.2803658Z ptp 36864 1 ena 2024-08-20T22:14:46.2804060Z pps_core 24576 1 ptp 2024-08-20T22:14:46.2804505Z aesni_intel 393216 0 2024-08-20T22:14:46.2805040Z i8042 45056 0 2024-08-20T22:14:46.2805577Z serio 28672 3 i8042 2024-08-20T22:14:46.2806229Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:46.2807007Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:46.2807665Z button 24576 0 2024-08-20T22:14:46.2808157Z sch_fq_codel 20480 17 2024-08-20T22:14:46.2808544Z dm_mod 188416 0 2024-08-20T22:14:46.2808906Z fuse 163840 1 2024-08-20T22:14:46.2809285Z dax 45056 1 dm_mod 2024-08-20T22:14:46.2809690Z configfs 57344 1 2024-08-20T22:14:46.2810070Z loop 36864 0 2024-08-20T22:14:46.2810436Z dmi_sysfs 20480 0 2024-08-20T22:14:46.2810813Z crc32_pclmul 16384 0 2024-08-20T22:14:46.2811183Z crc32c_intel 24576 0 2024-08-20T22:14:46.2811551Z efivarfs 24576 1 2024-08-20T22:14:46.2811919Z + modinfo nvidia 2024-08-20T22:14:46.2812679Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:46.2813380Z alias: char-major-195-* 2024-08-20T22:14:46.2813894Z version: 550.54.15 2024-08-20T22:14:46.2814256Z supported: external 2024-08-20T22:14:46.2814602Z license: NVIDIA 2024-08-20T22:14:46.2814984Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:46.2815463Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:46.2815926Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:46.2816397Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:46.2816902Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:46.2817384Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:46.2817895Z depends: i2c-core,drm 2024-08-20T22:14:46.2818259Z retpoline: Y 2024-08-20T22:14:46.2818560Z name: nvidia 2024-08-20T22:14:46.2819173Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:46.2819869Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:46.2820519Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:46.2821106Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:46.2821554Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:46.2821989Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:46.2822447Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:46.2822874Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:46.2823319Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:46.2823951Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:46.2824517Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:46.2824989Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:46.2825423Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:46.2825846Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:46.2826352Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:46.2826878Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:46.2827464Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:46.2828126Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:46.2828733Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:46.2829324Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:46.2829937Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:46.2830529Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:46.2831128Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:46.2831656Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:46.2832205Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:46.2832700Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:46.2833157Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:46.2833643Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:46.2834114Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:46.2834568Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:46.2835064Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:46.2835589Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:46.2836069Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:46.2836607Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:46.2837102Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:46.2837883Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:46.2838388Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:46.2838868Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:46.2839275Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:46.2839743Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:46.2840210Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:46.2840665Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:46.2841138Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:46.2841780Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:46.2842241Z parm: rm_firmware_active:charp 2024-08-20T22:14:46.2842652Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:14:46.2843072Z ++ command -v nvidia-smi 2024-08-20T22:14:46.2843479Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:14:46.2843853Z + set +e 2024-08-20T22:14:46.2844362Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:14:46.3090999Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:14:46.3091447Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:46.3091873Z + '[' 0 -ne 0 ']' 2024-08-20T22:14:46.3092347Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:14:46.3092812Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:14:46.3093611Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:14:46.3094326Z + set -e 2024-08-20T22:14:46.3094636Z + '[' 1 -eq 0 ']' 2024-08-20T22:14:46.3095227Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:14:46.3095924Z + post_install_nvidia_driver_common 2024-08-20T22:14:46.3098343Z + sudo modprobe nvidia 2024-08-20T22:14:46.4332993Z + echo 'After installing NVIDIA driver' 2024-08-20T22:14:46.4333451Z + lspci 2024-08-20T22:14:46.4333760Z After installing NVIDIA driver 2024-08-20T22:14:46.4439501Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:46.4440410Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:46.4441645Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:46.4442435Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:46.4443228Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:46.4444021Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:46.4444748Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:46.4445557Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:46.4446164Z + lsmod 2024-08-20T22:14:46.4472500Z Module Size Used by 2024-08-20T22:14:46.4472920Z xt_nat 16384 0 2024-08-20T22:14:46.4473420Z nvidia_modeset 1351680 0 2024-08-20T22:14:46.4473923Z video 65536 1 nvidia_modeset 2024-08-20T22:14:46.4474388Z wmi 36864 1 video 2024-08-20T22:14:46.4474930Z nvidia_uvm 4706304 0 2024-08-20T22:14:46.4475529Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:14:46.4476156Z drm 602112 1 nvidia 2024-08-20T22:14:46.4476738Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:14:46.4477470Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:14:46.4477989Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:14:46.4478407Z veth 36864 0 2024-08-20T22:14:46.4478788Z xt_conntrack 16384 1 2024-08-20T22:14:46.4479174Z nft_chain_nat 16384 3 2024-08-20T22:14:46.4479554Z xt_MASQUERADE 20480 1 2024-08-20T22:14:46.4480019Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:46.4480533Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:46.4481146Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:46.4481815Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:46.4482284Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:46.4482709Z xfrm_user 57344 1 2024-08-20T22:14:46.4483105Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:46.4483529Z xt_addrtype 16384 2 2024-08-20T22:14:46.4483899Z nft_compat 20480 4 2024-08-20T22:14:46.4484358Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:46.4484971Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:46.4485716Z br_netfilter 36864 0 2024-08-20T22:14:46.4486125Z bridge 307200 1 br_netfilter 2024-08-20T22:14:46.4486567Z stp 16384 1 bridge 2024-08-20T22:14:46.4486987Z llc 16384 2 bridge,stp 2024-08-20T22:14:46.4487398Z overlay 167936 0 2024-08-20T22:14:46.4487769Z tls 114688 0 2024-08-20T22:14:46.4488139Z nls_ascii 16384 1 2024-08-20T22:14:46.4488515Z nls_cp437 20480 1 2024-08-20T22:14:46.4488875Z vfat 24576 1 2024-08-20T22:14:46.4489245Z fat 86016 1 vfat 2024-08-20T22:14:46.4489637Z sunrpc 692224 1 2024-08-20T22:14:46.4490003Z ena 167936 0 2024-08-20T22:14:46.4490369Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:46.4490754Z ptp 36864 1 ena 2024-08-20T22:14:46.4491151Z pps_core 24576 1 ptp 2024-08-20T22:14:46.4491538Z aesni_intel 393216 0 2024-08-20T22:14:46.4491912Z i8042 45056 0 2024-08-20T22:14:46.4492289Z serio 28672 3 i8042 2024-08-20T22:14:46.4492712Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:46.4493315Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:46.4493984Z button 24576 0 2024-08-20T22:14:46.4494484Z sch_fq_codel 20480 17 2024-08-20T22:14:46.4494979Z dm_mod 188416 0 2024-08-20T22:14:46.4495616Z fuse 163840 1 2024-08-20T22:14:46.4496012Z dax 45056 1 dm_mod 2024-08-20T22:14:46.4496467Z configfs 57344 1 2024-08-20T22:14:46.4496839Z loop 36864 0 2024-08-20T22:14:46.4497197Z dmi_sysfs 20480 0 2024-08-20T22:14:46.4497571Z crc32_pclmul 16384 0 2024-08-20T22:14:46.4497942Z crc32c_intel 24576 0 2024-08-20T22:14:46.4498306Z efivarfs 24576 1 2024-08-20T22:14:46.4498671Z + modinfo nvidia 2024-08-20T22:14:46.4499396Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:46.4500095Z alias: char-major-195-* 2024-08-20T22:14:46.4500475Z version: 550.54.15 2024-08-20T22:14:46.4500822Z supported: external 2024-08-20T22:14:46.4501166Z license: NVIDIA 2024-08-20T22:14:46.4501545Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:46.4502025Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:46.4502488Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:46.4502955Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:46.4503459Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:46.4503944Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:46.4504448Z depends: i2c-core,drm 2024-08-20T22:14:46.4504812Z retpoline: Y 2024-08-20T22:14:46.4505123Z name: nvidia 2024-08-20T22:14:46.4505734Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:46.4506435Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:46.4507078Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:46.4507777Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:46.4508217Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:46.4508650Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:46.4509110Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:46.4509545Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:46.4509977Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:46.4510505Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:46.4511067Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:46.4511537Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:46.4511970Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:46.4512388Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:46.4512958Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:46.4513489Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:46.4514068Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:46.4514624Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:46.4515235Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:46.4515829Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:46.4516498Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:46.4517088Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:46.4517577Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:46.4518103Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:46.4518656Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:46.4519153Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:46.4519619Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:46.4520106Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:46.4520577Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:46.4521024Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:46.4521527Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:46.4522043Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:46.4522524Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:46.4523102Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:46.4523602Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:46.4524080Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:46.4524582Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:46.4525059Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:46.4525466Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:46.4525935Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:46.4526408Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:46.4526873Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:46.4527348Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:46.4527828Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:46.4528287Z parm: rm_firmware_active:charp 2024-08-20T22:14:46.4528697Z + set +e 2024-08-20T22:14:46.4529012Z + nvidia-smi 2024-08-20T22:14:46.4704196Z Tue Aug 20 22:14:46 2024 2024-08-20T22:14:46.4705596Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:46.4706871Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:14:46.4707758Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:46.4708562Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:14:46.4709456Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:14:46.4710148Z | | | MIG M. | 2024-08-20T22:14:46.4710702Z |=========================================+========================+======================| 2024-08-20T22:14:46.4960069Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:14:46.4960901Z | 0% 27C P8 21W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:14:46.4961557Z | | | N/A | 2024-08-20T22:14:46.4962266Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:46.4965423Z 2024-08-20T22:14:46.4966225Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:46.4967037Z | Processes: | 2024-08-20T22:14:46.4967748Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:14:46.4968442Z | ID ID Usage | 2024-08-20T22:14:46.4969007Z |=========================================================================================| 2024-08-20T22:14:46.4970839Z | No running processes found | 2024-08-20T22:14:46.4971623Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:46.7656167Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:14:46.7848097Z NVIDIA A10G 2024-08-20T22:14:46.7911585Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:46.7912315Z + '[' 0 -eq 0 ']' 2024-08-20T22:14:46.7912821Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:14:46.7913256Z + set -e 2024-08-20T22:14:46.7913799Z INFO: Ignoring allowed status 0 2024-08-20T22:14:46.7922083Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:14:46.7926268Z + sudo yum install -y yum-utils 2024-08-20T22:14:47.2363970Z Last metadata expiration check: 0:34:18 ago on Tue Aug 20 21:40:29 2024. 2024-08-20T22:14:47.2586011Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:14:47.2889127Z Dependencies resolved. 2024-08-20T22:14:47.3012829Z Nothing to do. 2024-08-20T22:14:47.3013236Z Complete! 2024-08-20T22:14:47.3780006Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:14:47.3781107Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:47.3782426Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:47.6526069Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:47.7207090Z + sudo yum install -y nvidia-docker2 2024-08-20T22:14:48.2304038Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:14:48.2523381Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:14:48.2821683Z Dependencies resolved. 2024-08-20T22:14:48.2946007Z Nothing to do. 2024-08-20T22:14:48.2946337Z Complete! 2024-08-20T22:14:48.3849392Z + sudo systemctl restart docker 2024-08-20T22:15:21.3407344Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T22:15:21.3643313Z Tue Aug 20 22:15:21 2024 2024-08-20T22:15:21.3644069Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:21.3648487Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:15:21.3649334Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:21.3650279Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:15:21.3651468Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:15:21.3652194Z | | | MIG M. | 2024-08-20T22:15:21.3652755Z |=========================================+========================+======================| 2024-08-20T22:15:21.3861521Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:15:21.3862436Z | 0% 27C P8 21W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:21.3863100Z | | | N/A | 2024-08-20T22:15:21.3863852Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:21.3866938Z 2024-08-20T22:15:21.3868061Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:21.3868713Z | Processes: | 2024-08-20T22:15:21.3869421Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:15:21.3870109Z | ID ID Usage | 2024-08-20T22:15:21.3870763Z |=========================================================================================| 2024-08-20T22:15:21.3871815Z | No running processes found | 2024-08-20T22:15:21.3872617Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:22.5460561Z Command completed after 1 attempt(s). 2024-08-20T22:15:22.5545810Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:22.5546587Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:22.5547273Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:15:22.5548005Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:22.5560540Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:22.5561038Z env: 2024-08-20T22:15:22.5561532Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:22.5561987Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:22.5562459Z ##[endgroup] 2024-08-20T22:15:23.0251533Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:15:23.4526571Z Collecting psutil==5.9.1 2024-08-20T22:15:23.4897732Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:15:23.5355420Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:15:23.5393791Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:15:23.6240873Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:15:23.7794444Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:15:23.9760082Z Prepare all required actions 2024-08-20T22:15:23.9760580Z Getting action download info 2024-08-20T22:15:24.1469417Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:15:24.3499180Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:15:24.5209986Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:15:24.5210475Z with: 2024-08-20T22:15:24.5210815Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:24.5211271Z s3-bucket: gha-artifacts 2024-08-20T22:15:24.5211606Z env: 2024-08-20T22:15:24.5211887Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:24.5212347Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:24.5212830Z ##[endgroup] 2024-08-20T22:15:24.5247295Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:24.5247757Z with: 2024-08-20T22:15:24.5248144Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:24.5248606Z s3-bucket: gha-artifacts 2024-08-20T22:15:24.5248972Z region: us-east-1 2024-08-20T22:15:24.5249284Z env: 2024-08-20T22:15:24.5249567Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:24.5250034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:24.5250524Z ##[endgroup] 2024-08-20T22:15:24.9961071Z (node:127430) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:24.9961768Z 2024-08-20T22:15:24.9962030Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:24.9962761Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:24.9963646Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:25.0748455Z Found 1 objects with prefix pytorch/pytorch/10479310721/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:15:25.0749552Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:32.6023261Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:32.6029791Z Artifact download has finished successfully 2024-08-20T22:15:32.6376859Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:15:32.6377297Z unzip -o artifacts.zip 2024-08-20T22:15:32.6386388Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:32.6386885Z env: 2024-08-20T22:15:32.6387164Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:32.6387739Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:32.6388227Z ##[endgroup] 2024-08-20T22:15:32.6703867Z Archive: artifacts.zip 2024-08-20T22:15:32.6705076Z creating: dist/ 2024-08-20T22:15:34.8059799Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:34.8060601Z creating: build/custom_test_artifacts/ 2024-08-20T22:15:34.8061305Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:15:34.8062055Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:15:34.8062901Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:34.8069370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:34.8070508Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:34.8071586Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:34.8072586Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:34.8073726Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:34.8074867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:34.8076500Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:34.8077605Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:34.8078643Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:34.8079969Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:34.8081370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:34.8083168Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:34.8084491Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:34.8085998Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:34.8087605Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:34.8088786Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:34.8089807Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:34.8129492Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:34.8169607Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:34.8171283Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:34.8216810Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:34.8218400Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:34.8219961Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:34.8221645Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:34.8223171Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:34.8224684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:34.8226481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:34.8228363Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:34.8230111Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:34.8231721Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:34.8233322Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:34.8234546Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:34.8235770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:34.8236966Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:34.8238582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:34.8292458Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:34.8352874Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:34.8354435Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:34.8355787Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:34.8356915Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:34.8358121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:34.8359334Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:15:34.8360597Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:34.8362142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:15:34.8363536Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:15:34.8364693Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:15:34.8365780Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:34.8366858Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:15:34.8367942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:34.8369016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:15:34.8370082Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:15:34.8385225Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:15:34.8528617Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:15:34.8529926Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:15:34.8531430Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:34.8533012Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:15:34.8534445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:15:34.8535789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:15:34.8537435Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:34.8538754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:15:34.8539903Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:34.8541039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:15:34.8542350Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:15:34.8559073Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:15:34.8643218Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:15:34.8644781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:34.8646121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:34.8647547Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:15:34.8648775Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:15:34.8649722Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:34.8650638Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:15:34.8651778Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:15:34.8652833Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:15:34.8653854Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:15:34.8773980Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:15:34.8837738Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:15:34.8838755Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:15:34.8839573Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:15:34.8840394Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:34.8846753Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:34.8847989Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:34.8849160Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:34.8850141Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:34.8851364Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:34.8852580Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:34.8854031Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:34.8855095Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:34.8856138Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:34.8857539Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:34.8858981Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:34.8860514Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:34.8861994Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:34.8863620Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:34.8865113Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:34.8866281Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:34.8867386Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:34.8906728Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:34.8948607Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:34.8951439Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:34.8994778Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:34.8996591Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:34.8998662Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:34.9000635Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:34.9002356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:34.9004166Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:34.9005929Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:34.9007726Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:34.9009326Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:34.9010678Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:34.9011958Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:34.9013189Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:34.9014414Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:34.9015610Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:34.9016811Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:34.9070845Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:34.9131340Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:34.9132900Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:34.9134197Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:34.9135311Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:34.9136501Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:34.9137933Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:15:34.9139305Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:15:34.9140830Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:15:34.9142325Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:15:34.9143622Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:15:34.9144932Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:15:34.9146050Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:15:34.9147152Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:15:34.9148344Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:15:34.9149447Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:15:34.9164494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:15:34.9230393Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:15:34.9231985Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:34.9233381Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:34.9234587Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:15:34.9235779Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:15:34.9236721Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:34.9237865Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:15:34.9239109Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:15:34.9240152Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:15:34.9241187Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:15:34.9290903Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:15:34.9291918Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:15:34.9292737Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:15:34.9293619Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:34.9299959Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:34.9301294Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:34.9302420Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:34.9303466Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:34.9304783Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:34.9306100Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:34.9307637Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:34.9308706Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:34.9309990Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:34.9311462Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:34.9312855Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:34.9314419Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:34.9315704Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:34.9317174Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:34.9318999Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:34.9320097Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:34.9321204Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:34.9360098Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:34.9399610Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:34.9401470Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:34.9446900Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:34.9448619Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:34.9450619Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:34.9452658Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:34.9454431Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:34.9456331Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:34.9458223Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:34.9460095Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:34.9461849Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:34.9463231Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:34.9464523Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:34.9465788Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:34.9467066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:34.9468384Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:34.9469657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:34.9522735Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:34.9583426Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:34.9585048Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:34.9586419Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:34.9587719Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:34.9589004Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:34.9590325Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:15:34.9591962Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:34.9593563Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:15:34.9595041Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:15:34.9596191Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:15:34.9597365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:34.9598947Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:15:34.9600201Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:34.9601440Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:15:34.9602609Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:15:34.9603843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:15:34.9725938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:15:34.9727461Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:15:34.9729009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:34.9730660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:15:34.9732400Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:15:34.9733946Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:15:34.9735252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:34.9736492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:15:34.9737994Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:34.9739235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:15:34.9740457Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:15:34.9756449Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:15:34.9812364Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:15:34.9814052Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:34.9815492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:34.9816764Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:15:34.9818014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:15:34.9818992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:34.9819979Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:15:34.9821228Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:15:34.9822577Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:15:34.9823492Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:15:34.9926642Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:15:34.9970441Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:15:34.9971717Z creating: build/lib/ 2024-08-20T22:15:35.0059862Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:15:35.0517729Z inflating: build/lib/libprotobuf.a 2024-08-20T22:15:35.0527221Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:15:35.0535742Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:15:35.0544383Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:15:35.0545032Z inflating: build/lib/libclog.a 2024-08-20T22:15:35.0563622Z inflating: build/lib/libnnpack.a 2024-08-20T22:15:35.0566152Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:15:35.0631654Z inflating: build/lib/libgtest.a 2024-08-20T22:15:35.0706876Z inflating: build/lib/libbenchmark.a 2024-08-20T22:15:35.0771678Z inflating: build/lib/libasmjit.a 2024-08-20T22:15:35.0779304Z inflating: build/lib/libittnotify.a 2024-08-20T22:15:35.0807100Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:15:35.0934805Z inflating: build/lib/libgloo.a 2024-08-20T22:15:35.0956036Z inflating: build/lib/libfmt.a 2024-08-20T22:15:35.1051059Z inflating: build/lib/libc10.so 2024-08-20T22:15:35.1052999Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:15:35.1054315Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:15:35.1560240Z inflating: build/lib/libprotoc.a 2024-08-20T22:15:35.1579564Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:15:35.1597758Z inflating: build/lib/libgmock.a 2024-08-20T22:15:35.1598377Z inflating: build/lib/libgtest_main.a 2024-08-20T22:15:35.1599452Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:15:35.1990121Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:15:36.2200788Z inflating: build/lib/libdnnl.a 2024-08-20T22:15:36.2775444Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:15:36.2833354Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:15:36.2834128Z inflating: build/lib/libgmock_main.a 2024-08-20T22:15:36.4100372Z inflating: build/lib/libfbgemm.a 2024-08-20T22:15:36.4358027Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:15:36.4867617Z inflating: build/lib/libkineto.a 2024-08-20T22:15:36.5052240Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:15:36.5094818Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:15:36.5789687Z inflating: build/lib/libonnx.a 2024-08-20T22:15:39.0330841Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:15:39.0335760Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:15:39.0340377Z inflating: build/lib/libshm.so 2024-08-20T22:15:41.1050155Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:15:41.1051272Z inflating: build/lib/libtorch.so 2024-08-20T22:15:41.1054071Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:15:41.9291425Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:15:42.1317321Z inflating: build/lib/libtorch_python.so 2024-08-20T22:15:42.1390582Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:15:42.1411724Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:15:42.1437712Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:15:42.1464409Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:15:42.1499054Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:15:42.1499651Z creating: build/bin/ 2024-08-20T22:15:42.1550615Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:15:42.1602299Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:15:42.1653856Z inflating: build/bin/c10_Device_test 2024-08-20T22:15:42.1713242Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:15:42.1768499Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:15:42.1817999Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:15:42.1869604Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:15:42.1924429Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:15:42.1980829Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:15:42.2037348Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:15:42.2108832Z inflating: build/bin/c10_cow_test 2024-08-20T22:15:42.2163318Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:15:42.2212595Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:15:42.2264492Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:15:42.2315087Z inflating: build/bin/c10_Half_test 2024-08-20T22:15:42.2372152Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:15:42.2427778Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:15:42.2478421Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:15:42.2534883Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:15:42.2586553Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:15:42.2638256Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:15:42.2687901Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:15:42.2740384Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:15:42.2796298Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:15:42.2847531Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:15:42.2904857Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:15:42.2961251Z inflating: build/bin/c10_complex_test 2024-08-20T22:15:42.3014373Z inflating: build/bin/c10_exception_test 2024-08-20T22:15:42.3065801Z inflating: build/bin/c10_flags_test 2024-08-20T22:15:42.3116347Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:15:42.3282758Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:15:42.3334185Z inflating: build/bin/c10_irange_test 2024-08-20T22:15:42.3388539Z inflating: build/bin/c10_lazy_test 2024-08-20T22:15:42.3446529Z inflating: build/bin/c10_logging_test 2024-08-20T22:15:42.3522885Z inflating: build/bin/c10_optional_test 2024-08-20T22:15:42.3586222Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:15:42.3640702Z inflating: build/bin/c10_registry_test 2024-08-20T22:15:42.3792839Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:15:42.3845960Z inflating: build/bin/c10_ssize_test 2024-08-20T22:15:42.3898148Z inflating: build/bin/c10_string_util_test 2024-08-20T22:15:42.3949340Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:15:42.4008485Z inflating: build/bin/c10_string_view_test 2024-08-20T22:15:42.4057824Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:15:42.4114329Z inflating: build/bin/c10_typeid_test 2024-08-20T22:15:42.4564613Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:15:42.5019998Z inflating: build/bin/protoc 2024-08-20T22:15:42.5073716Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:15:42.5127083Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:15:42.5181593Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:15:42.5235313Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:15:42.5288962Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:15:42.5342297Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:15:42.5391738Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:15:42.5445053Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:15:42.5781471Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:15:42.6129039Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:15:42.6490492Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:15:42.6543654Z inflating: build/bin/BackoffTest 2024-08-20T22:15:42.6597216Z inflating: build/bin/FileStoreTest 2024-08-20T22:15:42.6653673Z inflating: build/bin/TCPStoreTest 2024-08-20T22:15:42.6708100Z inflating: build/bin/HashStoreTest 2024-08-20T22:15:42.6721249Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:15:42.6775932Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:15:42.6780283Z inflating: build/bin/torch_shm_manager 2024-08-20T22:15:42.6783296Z inflating: build/bin/example_allreduce 2024-08-20T22:15:42.6839448Z inflating: build/bin/test_dist_autograd 2024-08-20T22:15:42.6908434Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:15:42.6911229Z inflating: build/bin/parallel_benchmark 2024-08-20T22:15:42.6979570Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:15:42.6988242Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:15:42.7360649Z inflating: build/bin/test_lazy 2024-08-20T22:15:42.8550968Z inflating: build/bin/test_api 2024-08-20T22:15:42.8625948Z inflating: build/bin/Dict_test 2024-08-20T22:15:42.8678606Z inflating: build/bin/Dimname_test 2024-08-20T22:15:42.8743763Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:15:42.8801622Z inflating: build/bin/NamedTensor_test 2024-08-20T22:15:42.8861341Z inflating: build/bin/apply_utils_test 2024-08-20T22:15:42.8921674Z inflating: build/bin/atest 2024-08-20T22:15:42.8985960Z inflating: build/bin/basic 2024-08-20T22:15:42.9040935Z inflating: build/bin/broadcast_test 2024-08-20T22:15:42.9092197Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:15:42.9151014Z inflating: build/bin/cpu_generator_test 2024-08-20T22:15:42.9204743Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:15:42.9298690Z inflating: build/bin/cpu_rng_test 2024-08-20T22:15:42.9349139Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:15:42.9399969Z inflating: build/bin/dlconvertor_test 2024-08-20T22:15:42.9458808Z inflating: build/bin/extension_backend_test 2024-08-20T22:15:42.9514609Z inflating: build/bin/half_test 2024-08-20T22:15:42.9611620Z inflating: build/bin/ivalue_test 2024-08-20T22:15:42.9662139Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:15:42.9716981Z inflating: build/bin/math_kernel_test 2024-08-20T22:15:42.9772763Z inflating: build/bin/memory_format_test 2024-08-20T22:15:42.9826977Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:15:42.9880551Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:15:42.9937718Z inflating: build/bin/native_test 2024-08-20T22:15:42.9988684Z inflating: build/bin/operator_name_test 2024-08-20T22:15:43.0041772Z inflating: build/bin/operators_test 2024-08-20T22:15:43.0094601Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:15:43.0161858Z inflating: build/bin/pow_test 2024-08-20T22:15:43.0219530Z inflating: build/bin/quantized_test 2024-08-20T22:15:43.0270199Z inflating: build/bin/reduce_ops_test 2024-08-20T22:15:43.0321893Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:15:43.0378948Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:15:43.0438291Z inflating: build/bin/scalar_test 2024-08-20T22:15:43.0490711Z inflating: build/bin/StorageUtils_test 2024-08-20T22:15:43.0544054Z inflating: build/bin/stride_properties_test 2024-08-20T22:15:43.0624189Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:15:43.0679012Z inflating: build/bin/test_parallel 2024-08-20T22:15:43.0682195Z inflating: build/bin/thread_init_test 2024-08-20T22:15:43.0738581Z inflating: build/bin/type_ptr_test 2024-08-20T22:15:43.0799102Z inflating: build/bin/type_test 2024-08-20T22:15:43.0852128Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:15:43.0853573Z inflating: build/bin/verify_api_visibility 2024-08-20T22:15:43.0923874Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:15:43.0976221Z inflating: build/bin/weakref_test 2024-08-20T22:15:43.1028070Z inflating: build/bin/wrapdim_test 2024-08-20T22:15:43.1080376Z inflating: build/bin/xla_tensor_test 2024-08-20T22:15:43.1142202Z inflating: build/bin/IListRef_test 2024-08-20T22:15:43.1249508Z inflating: build/bin/List_test 2024-08-20T22:15:43.1316699Z inflating: build/bin/KernelFunction_test 2024-08-20T22:15:43.1440534Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:15:43.1538538Z inflating: build/bin/kernel_function_test 2024-08-20T22:15:43.1667894Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:15:43.1772458Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:15:43.1834578Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:15:43.1931960Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:15:43.1984343Z inflating: build/bin/CppSignature_test 2024-08-20T22:15:43.2033815Z inflating: build/bin/op_allowlist_test 2024-08-20T22:15:43.2090369Z inflating: build/bin/backend_fallback_test 2024-08-20T22:15:43.2394781Z inflating: build/bin/op_registration_test 2024-08-20T22:15:43.2459058Z inflating: build/bin/inline_container_test 2024-08-20T22:15:43.2513535Z inflating: build/bin/cuda_apply_test 2024-08-20T22:15:43.2566199Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:15:43.2626143Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:15:43.2681342Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:15:43.2752116Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:15:43.2811939Z inflating: build/bin/cuda_complex_test 2024-08-20T22:15:43.2862467Z inflating: build/bin/cuda_device_test 2024-08-20T22:15:43.2920958Z inflating: build/bin/cuda_cub_test 2024-08-20T22:15:43.2972387Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:15:43.3037607Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:15:43.3095894Z inflating: build/bin/cuda_generator_test 2024-08-20T22:15:43.3146363Z inflating: build/bin/cuda_half_test 2024-08-20T22:15:43.3198500Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:15:43.3248568Z inflating: build/bin/cuda_optional_test 2024-08-20T22:15:43.3300987Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:15:43.3354521Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:15:43.3404719Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:15:43.3466132Z inflating: build/bin/cuda_stream_test 2024-08-20T22:15:43.3516042Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:15:43.3568585Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:15:43.3583220Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:15:43.3649661Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:15:43.3707574Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:15:43.3771993Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:15:43.3834940Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:15:43.4694371Z inflating: build/bin/test_tensorexpr 2024-08-20T22:15:43.5280529Z inflating: build/bin/test_jit 2024-08-20T22:15:43.5281111Z creating: .additional_ci_files/ 2024-08-20T22:15:43.5342668Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:15:43.5585386Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:15:43.5622164Z ##[group]Run rm artifacts.zip 2024-08-20T22:15:43.5622555Z rm artifacts.zip 2024-08-20T22:15:43.5631752Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:43.5632239Z env: 2024-08-20T22:15:43.5632519Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:43.5632982Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:43.5633698Z ##[endgroup] 2024-08-20T22:15:43.6714472Z ##[group]Run df -H 2024-08-20T22:15:43.6714788Z df -H 2024-08-20T22:15:43.6723541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:43.6724043Z env: 2024-08-20T22:15:43.6724326Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:43.6724779Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:43.6725251Z ##[endgroup] 2024-08-20T22:15:43.6795532Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:15:43.6796248Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:15:43.6796774Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:15:43.6797250Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:15:43.6797698Z /dev/nvme0n1p1 161G 76G 86G 48% / 2024-08-20T22:15:43.6798378Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:15:43.6798873Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:15:43.6799376Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:15:43.6859020Z Prepare all required actions 2024-08-20T22:15:43.6859656Z Getting action download info 2024-08-20T22:15:43.8330545Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:15:43.8331009Z with: 2024-08-20T22:15:43.8331272Z env: 2024-08-20T22:15:43.8331548Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:43.8331995Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:43.8332474Z ##[endgroup] 2024-08-20T22:15:43.8366140Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:43.8366581Z with: 2024-08-20T22:15:43.8366845Z name: td_results 2024-08-20T22:15:43.8367167Z s3-bucket: gha-artifacts 2024-08-20T22:15:43.8367518Z region: us-east-1 2024-08-20T22:15:43.8367823Z env: 2024-08-20T22:15:43.8368090Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:43.8368541Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:43.8369018Z ##[endgroup] 2024-08-20T22:15:44.3121742Z (node:127451) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:44.3122536Z 2024-08-20T22:15:44.3122809Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:44.3123576Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:44.3124439Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:44.3715259Z Found 0 objects with prefix pytorch/pytorch/10479310721/td_results/ 2024-08-20T22:15:44.3721921Z Artifact download has finished successfully 2024-08-20T22:15:44.4055056Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:15:44.4055545Z mkdir -p .additional_ci_files 2024-08-20T22:15:44.4056109Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:15:44.4066348Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:44.4066842Z env: 2024-08-20T22:15:44.4067124Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:44.4067686Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:44.4068185Z ##[endgroup] 2024-08-20T22:15:44.4147953Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:15:44.4160796Z ##[error]Process completed with exit code 1. 2024-08-20T22:15:44.4195240Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:15:44.4195732Z .github/scripts/parse_ref.py 2024-08-20T22:15:44.4204478Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:44.4204824Z env: 2024-08-20T22:15:44.4205113Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:44.4205576Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:44.4206061Z ##[endgroup] 2024-08-20T22:15:44.4502929Z Prepare all required actions 2024-08-20T22:15:44.4541714Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:15:44.4542175Z with: 2024-08-20T22:15:44.4542749Z github-token: *** 2024-08-20T22:15:44.4543061Z env: 2024-08-20T22:15:44.4543333Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:44.4543798Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:44.4544500Z ##[endgroup] 2024-08-20T22:15:44.4562358Z ##[group]Run set -eux 2024-08-20T22:15:44.4562694Z set -eux 2024-08-20T22:15:44.4563280Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:15:44.4571926Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:44.4572419Z env: 2024-08-20T22:15:44.4572698Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:44.4573151Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:44.4573792Z GITHUB_TOKEN: *** 2024-08-20T22:15:44.4574100Z ##[endgroup] 2024-08-20T22:15:44.4604669Z + python3 .github/scripts/get_workflow_job_id.py 10479310721 i-0fa2b5ff9d3cb2975 2024-08-20T22:15:45.1389773Z setting job-id=29026300057 2024-08-20T22:15:45.1390844Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:45.1604534Z Prepare all required actions 2024-08-20T22:15:45.1604999Z Getting action download info 2024-08-20T22:15:45.2760130Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:15:45.2760584Z with: 2024-08-20T22:15:45.2761049Z github-token: *** 2024-08-20T22:15:45.2767526Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:15:45.2774828Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:45.2775741Z env: 2024-08-20T22:15:45.2776015Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:45.2776475Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:45.2776958Z ##[endgroup] 2024-08-20T22:15:45.2819367Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:15:45.2819909Z with: 2024-08-20T22:15:45.2820178Z shell: bash 2024-08-20T22:15:45.2820479Z timeout_minutes: 10 2024-08-20T22:15:45.2820797Z max_attempts: 5 2024-08-20T22:15:45.2821111Z retry_wait_seconds: 30 2024-08-20T22:15:45.2822203Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:45.2823350Z polling_interval_seconds: 1 2024-08-20T22:15:45.2823723Z warning_on_retry: true 2024-08-20T22:15:45.2824115Z continue_on_error: false 2024-08-20T22:15:45.2824461Z env: 2024-08-20T22:15:45.2824915Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:45.2825352Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:45.2825991Z GITHUB_TOKEN: *** 2024-08-20T22:15:45.2826305Z ##[endgroup] 2024-08-20T22:15:45.5205354Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:45.9048224Z Collecting requests==2.27.1 2024-08-20T22:15:45.9391158Z Downloading requests-2.27.1-py2.py3-none-any.whl.metadata (5.0 kB) 2024-08-20T22:15:45.9847475Z Collecting pyyaml==6.0.1 2024-08-20T22:15:45.9877329Z Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) 2024-08-20T22:15:46.0316869Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.27.1) 2024-08-20T22:15:46.0345033Z Downloading urllib3-1.26.19-py2.py3-none-any.whl.metadata (49 kB) 2024-08-20T22:15:46.0525879Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T22:15:46.1189468Z Collecting charset-normalizer~=2.0.0 (from requests==2.27.1) 2024-08-20T22:15:46.1217624Z Downloading charset_normalizer-2.0.12-py3-none-any.whl.metadata (11 kB) 2024-08-20T22:15:46.1277986Z Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (3.7) 2024-08-20T22:15:46.1356615Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:15:46.1445142Z Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (724 kB) 2024-08-20T22:15:46.1607668Z ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 725.0/725.0 kB 46.3 MB/s eta 0:00:00 2024-08-20T22:15:46.1636049Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:15:46.1701469Z Downloading urllib3-1.26.19-py2.py3-none-any.whl (143 kB) 2024-08-20T22:15:46.2417873Z Installing collected packages: urllib3, pyyaml, charset-normalizer, requests 2024-08-20T22:15:46.2419303Z Attempting uninstall: urllib3 2024-08-20T22:15:46.2439352Z Found existing installation: urllib3 2.2.2 2024-08-20T22:15:46.2485148Z Uninstalling urllib3-2.2.2: 2024-08-20T22:15:46.2494412Z Successfully uninstalled urllib3-2.2.2 2024-08-20T22:15:46.3640324Z Attempting uninstall: charset-normalizer 2024-08-20T22:15:46.3675475Z Found existing installation: charset-normalizer 3.3.2 2024-08-20T22:15:46.3717481Z Uninstalling charset-normalizer-3.3.2: 2024-08-20T22:15:46.3740062Z Successfully uninstalled charset-normalizer-3.3.2 2024-08-20T22:15:46.4002656Z Attempting uninstall: requests 2024-08-20T22:15:46.4029666Z Found existing installation: requests 2.32.3 2024-08-20T22:15:46.4059891Z Uninstalling requests-2.32.3: 2024-08-20T22:15:46.4066215Z Successfully uninstalled requests-2.32.3 2024-08-20T22:15:46.4426452Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:15:46.4427877Z conda 24.7.1 requires requests<3,>=2.28.0, but you have requests 2.27.1 which is incompatible. 2024-08-20T22:15:46.4428949Z Successfully installed charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 urllib3-1.26.19 2024-08-20T22:15:47.3382279Z Command completed after 1 attempt(s). 2024-08-20T22:15:47.3442001Z ##[group]Run set -x 2024-08-20T22:15:47.3442340Z set -x 2024-08-20T22:15:47.3442638Z  2024-08-20T22:15:47.3443182Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:47.3443852Z # in runner workspace 2024-08-20T22:15:47.3444385Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:15:47.3454417Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:47.3454915Z env: 2024-08-20T22:15:47.3455191Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:47.3455640Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:47.3456138Z ##[endgroup] 2024-08-20T22:15:47.3486538Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:15:47.3729902Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:47.3730456Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:47.3730934Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:15:47.3731345Z  2024-08-20T22:15:47.3731890Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:47.3732584Z # in runner workspace 2024-08-20T22:15:47.3733167Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:15:47.3733820Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:15:47.3734381Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:15:47.3741864Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:15:47.3749271Z  --selected-test-configs "" \ 2024-08-20T22:15:47.3749741Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:15:47.3750162Z  --tag "${TAG}" \ 2024-08-20T22:15:47.3750571Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:15:47.3751020Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:15:47.3751454Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:15:47.3760634Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:47.3761151Z env: 2024-08-20T22:15:47.3761441Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:47.3761904Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:47.3762590Z GITHUB_TOKEN: *** 2024-08-20T22:15:47.3763451Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:47.3764409Z PR_NUMBER: 2024-08-20T22:15:47.3764748Z TAG: ciflow/inductor/133712 2024-08-20T22:15:47.3765128Z EVENT_NAME: push 2024-08-20T22:15:47.3765450Z SCHEDULE: 2024-08-20T22:15:47.3765754Z HEAD_BRANCH: 2024-08-20T22:15:47.3766067Z ##[endgroup] 2024-08-20T22:15:47.3796018Z Workflow: inductor-periodic 2024-08-20T22:15:47.3797113Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:47.5988033Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:15:47.7688111Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:15:47.7688757Z echo "Filtered matrix:" 2024-08-20T22:15:47.7695426Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:15:47.7702198Z  2024-08-20T22:15:47.7702473Z echo 2024-08-20T22:15:47.7702853Z echo "Is the current job unstable? False" 2024-08-20T22:15:47.7703301Z  2024-08-20T22:15:47.7703573Z echo 2024-08-20T22:15:47.7703938Z echo "Is keep-going label set? False" 2024-08-20T22:15:47.7704376Z  2024-08-20T22:15:47.7704645Z echo 2024-08-20T22:15:47.7704958Z echo "Renabled issues? " 2024-08-20T22:15:47.7713831Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:47.7714324Z env: 2024-08-20T22:15:47.7714601Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:47.7715056Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:47.7715534Z ##[endgroup] 2024-08-20T22:15:47.7745056Z Filtered matrix: 2024-08-20T22:15:47.7752790Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:15:47.7759386Z 2024-08-20T22:15:47.7759555Z Is the current job unstable? False 2024-08-20T22:15:47.7760016Z 2024-08-20T22:15:47.7760384Z Is keep-going label set? False 2024-08-20T22:15:47.7760644Z 2024-08-20T22:15:47.7760781Z Renabled issues? 2024-08-20T22:15:47.7802804Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:47.7803515Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:47.7811691Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:47.7812181Z env: 2024-08-20T22:15:47.7812458Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:47.7812917Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:47.7813399Z JOB_TIMEOUT: 240 2024-08-20T22:15:47.7813701Z ##[endgroup] 2024-08-20T22:15:47.7895054Z ##[group]Run set -x 2024-08-20T22:15:47.7895451Z set -x 2024-08-20T22:15:47.7895748Z  2024-08-20T22:15:47.7896104Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:15:47.7896641Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:15:47.7897212Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:15:47.7897721Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:15:47.7898134Z else 2024-08-20T22:15:47.7898472Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:47.7898895Z fi 2024-08-20T22:15:47.7899173Z  2024-08-20T22:15:47.7899639Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:15:47.7900381Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:15:47.7901052Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:15:47.7901640Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:15:47.7902097Z container_name=$(docker run \ 2024-08-20T22:15:47.7902518Z  ${GPU_FLAG:-} \ 2024-08-20T22:15:47.7902898Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:15:47.7903307Z  -e PR_NUMBER \ 2024-08-20T22:15:47.7903671Z  -e GITHUB_ACTIONS \ 2024-08-20T22:15:47.7904068Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:15:47.7904483Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:15:47.7904896Z  -e GITHUB_JOB \ 2024-08-20T22:15:47.7905288Z  -e GITHUB_RUN_ID \ 2024-08-20T22:15:47.7905678Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:15:47.7906096Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:15:47.7906502Z  -e JOB_ID \ 2024-08-20T22:15:47.7906845Z  -e JOB_NAME \ 2024-08-20T22:15:47.7907202Z  -e BASE_SHA \ 2024-08-20T22:15:47.7907682Z  -e BRANCH \ 2024-08-20T22:15:47.7908014Z  -e SHA1 \ 2024-08-20T22:15:47.7908367Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:15:47.7908772Z  -e IN_WHEEL_TEST \ 2024-08-20T22:15:47.7909159Z  -e SHARD_NUMBER \ 2024-08-20T22:15:47.7909525Z  -e TEST_CONFIG \ 2024-08-20T22:15:47.7909896Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:15:47.7910297Z  -e REENABLED_ISSUES \ 2024-08-20T22:15:47.7910711Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:15:47.7911130Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:15:47.7911533Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:15:47.7911926Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:15:47.7912298Z  -e NO_TD \ 2024-08-20T22:15:47.7912635Z  -e TD_DISTRIBUTED \ 2024-08-20T22:15:47.7913016Z  -e PR_LABELS \ 2024-08-20T22:15:47.7913429Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:15:47.7913880Z  -e SCCACHE_BUCKET \ 2024-08-20T22:15:47.7914277Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:15:47.7914686Z  -e XLA_CUDA \ 2024-08-20T22:15:47.7915101Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:15:47.7915651Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:15:47.7916175Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:15:47.7916694Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:15:47.7917336Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:15:47.7917783Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:15:47.7918223Z  -e DASHBOARD_TAG \ 2024-08-20T22:15:47.7918701Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:15:47.7919263Z  --security-opt seccomp=unconfined \ 2024-08-20T22:15:47.7919721Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:15:47.7920118Z  --ipc=host \ 2024-08-20T22:15:47.7920485Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:15:47.7920881Z  --tty \ 2024-08-20T22:15:47.7921190Z  --detach \ 2024-08-20T22:15:47.7921667Z  --name="${container_name}" \ 2024-08-20T22:15:47.7922086Z  --user jenkins \ 2024-08-20T22:15:47.7922573Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:15:47.7923122Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:15:47.7923568Z  "${DOCKER_IMAGE}" 2024-08-20T22:15:47.7923917Z ) 2024-08-20T22:15:47.7924323Z # Propagate download.pytorch.org IP to container 2024-08-20T22:15:47.7925236Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:15:47.7926191Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:15:47.7927094Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:15:47.7935571Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:47.7935916Z env: 2024-08-20T22:15:47.7936197Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:47.7936655Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:47.7937547Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:47.7938054Z PR_NUMBER: 2024-08-20T22:15:47.7938384Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:15:47.7938823Z GITHUB_WORKFLOW: inductor-periodic 2024-08-20T22:15:47.7939228Z GITHUB_JOB: test 2024-08-20T22:15:47.7939543Z GITHUB_RUN_ID: 10479310721 2024-08-20T22:15:47.7939912Z GITHUB_RUN_NUMBER: 54565 2024-08-20T22:15:47.7940264Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:15:47.7940603Z JOB_ID: 29026300057 2024-08-20T22:15:47.7941462Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:47.7942383Z BRANCH: 2024-08-20T22:15:47.7942729Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:47.7943254Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:47.7943758Z TEST_CONFIG: dynamic_aot_eager_torchbench 2024-08-20T22:15:47.7944188Z SHARD_NUMBER: 2 2024-08-20T22:15:47.7944503Z NUM_TEST_SHARDS: 2 2024-08-20T22:15:47.7944828Z REENABLED_ISSUES: 2024-08-20T22:15:47.7945189Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:15:47.7945597Z VERBOSE_TEST_LOGS: False 2024-08-20T22:15:47.7945959Z TEST_SHOWLOCALS: False 2024-08-20T22:15:47.7946300Z NO_TEST_TIMEOUT: False 2024-08-20T22:15:47.7946636Z NO_TD: False 2024-08-20T22:15:47.7946947Z TD_DISTRIBUTED: False 2024-08-20T22:15:47.7947422Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:15:47.7947931Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-08-20T22:15:47.7948357Z SHM_SIZE: 2g 2024-08-20T22:15:47.7949436Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:47.7950594Z XLA_CUDA: 2024-08-20T22:15:47.7951069Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:15:47.7951684Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:15:47.7952118Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:15:47.7952528Z DASHBOARD_TAG: 2024-08-20T22:15:47.7953041Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:15:47.7953419Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:15:47.7953950Z ##[endgroup] 2024-08-20T22:15:47.7982217Z + [[ dynamic_aot_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:15:47.7983042Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:15:47.7983562Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:47.7991213Z +++ nproc --ignore=2 2024-08-20T22:15:47.8016028Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310721 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:55.0321134Z + container_name=b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-20T22:15:55.0323514Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:15:55.0325338Z + docker exec -i b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:15:55.1660506Z + echo DOCKER_CONTAINER_ID=b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-20T22:15:55.1664797Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:55.1667873Z + docker exec -t b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:15:55.5698594Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:15:55.8932880Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:15:55.8936186Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:15:55.8939535Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:15:55.8941581Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:15:55.8945654Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:15:55.8950672Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:15:55.8967185Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:15:55.8980787Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:15:55.8998109Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:15:55.9383775Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:15:56.5125012Z Installing collected packages: torch 2024-08-20T22:16:06.6257783Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:16:06.6259551Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:16:06.6260273Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:16:06.7074125Z + export TERM=vt100 2024-08-20T22:16:06.7074506Z + TERM=vt100 2024-08-20T22:16:06.7077492Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:16:06.7093317Z + source .ci/pytorch/common.sh 2024-08-20T22:16:06.7096839Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:16:06.7107840Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:16:06.7109191Z +++ declare -f -t trap_add 2024-08-20T22:16:06.7115379Z ++ set -ex 2024-08-20T22:16:06.7116061Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:06.7116606Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:16:06.7117204Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:16:06.7119995Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:16:06.7158476Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:16:06.7158941Z + trap_add cleanup_workspace EXIT 2024-08-20T22:16:06.7159375Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:16:06.7159747Z + shift 2024-08-20T22:16:06.7160032Z + for trap_add_name in "$@" 2024-08-20T22:16:06.7167069Z +++ trap -p EXIT 2024-08-20T22:16:06.7170728Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:16:06.7171110Z +++ extract_trap_cmd 2024-08-20T22:16:06.7171493Z +++ printf '%s\n' '' 2024-08-20T22:16:06.7171882Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:16:06.7174040Z + trap -- ' 2024-08-20T22:16:06.7174399Z cleanup_workspace' EXIT 2024-08-20T22:16:06.7174871Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:16:07.4413673Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:16:07.4436710Z + echo 'Environment variables:' 2024-08-20T22:16:07.4437133Z Environment variables: 2024-08-20T22:16:07.4437646Z + env 2024-08-20T22:16:07.4460527Z INSTALLED_DB=yes 2024-08-20T22:16:07.4461301Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:07.4461824Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:07.4462260Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:07.4462880Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:07.4463449Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:07.4463952Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:07.4464486Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:07.4465155Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:07.4465694Z HOSTNAME=b5ad0927b399 2024-08-20T22:16:07.4466547Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4467423Z GITHUB_ACTION=__self 2024-08-20T22:16:07.4467779Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:07.4471881Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:07.4476074Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:07.4476588Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:07.4476937Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:07.4477310Z TEST_CONFIG=dynamic_aot_eager_torchbench 2024-08-20T22:16:07.4477747Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:07.4478269Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:07.4478734Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:07.4479319Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:07.4479733Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:07.4480150Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:07.4480633Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:07.4481125Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:07.4481591Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:07.4481921Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:07.4482299Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:07.4482695Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4483135Z XLA_CUDA= 2024-08-20T22:16:07.4483623Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:07.4485992Z *** 2024-08-20T22:16:07.4486299Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:07.4486738Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:07.4487110Z GITHUB_ACTIONS=true 2024-08-20T22:16:07.4487439Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:07.4487966Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:07.4488547Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:07.4489056Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4489605Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:07.4490124Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4491167Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:07.4492026Z UCC_HOME=/usr 2024-08-20T22:16:07.4492373Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:07.4492794Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:07.4493151Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:07.4493636Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:07.4494168Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:07.4494666Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:07.4495062Z SHARD_NUMBER=2 2024-08-20T22:16:07.4495377Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:07.4495748Z HOME=/var/lib/jenkins 2024-08-20T22:16:07.4496130Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:07.4496598Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:07.4497097Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:07.4497693Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:07.4498133Z CUDA_VERSION=12.1.1 2024-08-20T22:16:07.4498572Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:07.4499044Z NUM_TEST_SHARDS=2 2024-08-20T22:16:07.4499352Z UCX_HOME=/usr 2024-08-20T22:16:07.4499915Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:07.4501122Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4502788Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:07.4504402Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4505660Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:07.4506377Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:07.4506711Z DASHBOARD_TAG= 2024-08-20T22:16:07.4507022Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:07.4507584Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:07.4508245Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:07.4509232Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4510132Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:07.4510541Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:07.4510912Z PR_NUMBER= 2024-08-20T22:16:07.4511202Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:07.4511537Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:07.4511982Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:07.4512431Z TERM=vt100 2024-08-20T22:16:07.4512770Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:07.4513273Z INSTALLED_VISION=yes 2024-08-20T22:16:07.4513584Z BRANCH= 2024-08-20T22:16:07.4513867Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:07.4514264Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:07.4514688Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:07.4515431Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:07.4516179Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:07.4516663Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:07.4517120Z REENABLED_ISSUES= 2024-08-20T22:16:07.4517413Z SHLVL=1 2024-08-20T22:16:07.4517671Z MAX_JOBS=14 2024-08-20T22:16:07.4517989Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:07.4518339Z NVARCH=x86_64 2024-08-20T22:16:07.4518632Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:07.4519086Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4519613Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:07.4520093Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:07.4520762Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:07.4521324Z GITHUB_JOB=test 2024-08-20T22:16:07.4521729Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:07.4522273Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:07.4522779Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:07.4523108Z TD_DISTRIBUTED=False 2024-08-20T22:16:07.4523490Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:07.4523919Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:07.4524339Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:07.4524697Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:07.4525046Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:07.4525389Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:07.4526409Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:07.4527454Z GITHUB_BASE_REF= 2024-08-20T22:16:07.4527786Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:07.4528148Z CI=true 2024-08-20T22:16:07.4528475Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:07.4528886Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:07.4529254Z JOB_ID=29026300057 2024-08-20T22:16:07.4529571Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:07.4529899Z GITHUB_HEAD_REF= 2024-08-20T22:16:07.4530204Z GITHUB_ACTION_REF= 2024-08-20T22:16:07.4530680Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:07.4531149Z TEST_SHOWLOCALS=False 2024-08-20T22:16:07.4531544Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:07.4531939Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:07.4532843Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4533649Z NO_TD=False 2024-08-20T22:16:07.4533953Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:07.4534313Z _=/usr/bin/env 2024-08-20T22:16:07.4534795Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:16:07.4692446Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:16:07.4693608Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:16:07.4694693Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:16:07.4695767Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:16:07.4696371Z + BUILD_DIR=build 2024-08-20T22:16:07.4696704Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:16:07.4697091Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:16:07.4697430Z + SHARD_NUMBER=2 2024-08-20T22:16:07.4697741Z + NUM_TEST_SHARDS=2 2024-08-20T22:16:07.4698174Z + export VALGRIND=ON 2024-08-20T22:16:07.4698575Z + VALGRIND=ON 2024-08-20T22:16:07.4699064Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:16:07.4699549Z + [[ 0 == \1 ]] 2024-08-20T22:16:07.4699846Z + [[ False == \1 ]] 2024-08-20T22:16:07.4700521Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:16:07.4701052Z ++ realpath build/custom_test_artifacts 2024-08-20T22:16:07.4718985Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:16:07.4719686Z + [[ -n '' ]] 2024-08-20T22:16:07.4720060Z + echo 'Environment variables' 2024-08-20T22:16:07.4720492Z Environment variables 2024-08-20T22:16:07.4720810Z + env 2024-08-20T22:16:07.4728435Z INSTALLED_DB=yes 2024-08-20T22:16:07.4729016Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:07.4729563Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:07.4730094Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:07.4730872Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:07.4731614Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:07.4732285Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:07.4732900Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:07.4733754Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:07.4734408Z HOSTNAME=b5ad0927b399 2024-08-20T22:16:07.4735357Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4736242Z GITHUB_ACTION=__self 2024-08-20T22:16:07.4736705Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:07.4740798Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:07.4744601Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:07.4745118Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:07.4745467Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:07.4745846Z TEST_CONFIG=dynamic_aot_eager_torchbench 2024-08-20T22:16:07.4746293Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:07.4746769Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:07.4747316Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:07.4747768Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:07.4748177Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:07.4748565Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:07.4749010Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:07.4749512Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:07.4749972Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:07.4750296Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:07.4750693Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:07.4751090Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4751524Z XLA_CUDA= 2024-08-20T22:16:07.4751978Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:07.4752366Z *** 2024-08-20T22:16:07.4752660Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:07.4753269Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:07.4753631Z GITHUB_ACTIONS=true 2024-08-20T22:16:07.4754066Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:07.4754578Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:07.4755111Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:07.4755587Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4756110Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:07.4756592Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4757506Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:07.4758409Z UCC_HOME=/usr 2024-08-20T22:16:07.4758774Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:07.4759173Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:07.4759510Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:07.4759976Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:07.4760479Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:07.4760951Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:07.4761319Z SHARD_NUMBER=2 2024-08-20T22:16:07.4761627Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:07.4761981Z HOME=/var/lib/jenkins 2024-08-20T22:16:07.4762343Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:07.4762782Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:07.4763258Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:07.4763808Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:07.4764214Z CUDA_VERSION=12.1.1 2024-08-20T22:16:07.4764740Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:07.4765304Z NUM_TEST_SHARDS=2 2024-08-20T22:16:07.4765625Z UCX_HOME=/usr 2024-08-20T22:16:07.4766157Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:07.4767226Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4768686Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:07.4770098Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4771189Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:07.4771840Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:07.4772172Z DASHBOARD_TAG= 2024-08-20T22:16:07.4772477Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:07.4772934Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:07.4773477Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:07.4774468Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4775366Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:07.4775769Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:07.4776148Z PR_NUMBER= 2024-08-20T22:16:07.4776432Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:07.4776758Z VALGRIND=ON 2024-08-20T22:16:07.4777051Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:07.4777502Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:07.4777961Z TERM=vt100 2024-08-20T22:16:07.4778311Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:07.4778723Z INSTALLED_VISION=yes 2024-08-20T22:16:07.4779038Z BRANCH= 2024-08-20T22:16:07.4779322Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:07.4779717Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:07.4780145Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:07.4780906Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:07.4781650Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:07.4782127Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:07.4782594Z REENABLED_ISSUES= 2024-08-20T22:16:07.4782893Z SHLVL=1 2024-08-20T22:16:07.4783146Z MAX_JOBS=14 2024-08-20T22:16:07.4783594Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:07.4783946Z NVARCH=x86_64 2024-08-20T22:16:07.4784241Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:07.4784690Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:07.4785225Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:07.4785742Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:07.4786490Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:07.4787053Z GITHUB_JOB=test 2024-08-20T22:16:07.4787584Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:07.4788179Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:07.4788853Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:07.4789186Z TD_DISTRIBUTED=False 2024-08-20T22:16:07.4789572Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:07.4790003Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:07.4790416Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:07.4790778Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:07.4791137Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:07.4791486Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:07.4792503Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:07.4793547Z GITHUB_BASE_REF= 2024-08-20T22:16:07.4793876Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:07.4794243Z CI=true 2024-08-20T22:16:07.4794557Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:07.4794975Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:07.4795348Z JOB_ID=29026300057 2024-08-20T22:16:07.4795660Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:07.4795992Z GITHUB_HEAD_REF= 2024-08-20T22:16:07.4796318Z GITHUB_ACTION_REF= 2024-08-20T22:16:07.4796891Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:07.4797419Z TEST_SHOWLOCALS=False 2024-08-20T22:16:07.4797824Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:07.4798231Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:07.4799140Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ece555b7-d7c6-4697-a98c-7f5e00c42464 2024-08-20T22:16:07.4799933Z NO_TD=False 2024-08-20T22:16:07.4800233Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:07.4800596Z _=/usr/bin/env 2024-08-20T22:16:07.4800948Z + echo 'Testing pytorch' 2024-08-20T22:16:07.4801295Z Testing pytorch 2024-08-20T22:16:07.4801625Z + export LANG=C.UTF-8 2024-08-20T22:16:07.4801967Z + LANG=C.UTF-8 2024-08-20T22:16:07.4807793Z + PR_NUMBER= 2024-08-20T22:16:07.4808309Z + [[ dynamic_aot_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-08-20T22:16:07.4809098Z + [[ dynamic_aot_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:16:07.4809838Z + [[ dynamic_aot_eager_torchbench == \s\l\o\w ]] 2024-08-20T22:16:07.4810679Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:16:07.4811597Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:16:07.4812208Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:07.4812685Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:07.4813168Z + [[ dynamic_aot_eager_torchbench == *crossref* ]] 2024-08-20T22:16:07.4813774Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:07.4814407Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:16:07.4815062Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:07.4815627Z + pip_install --user ninja==1.10.2 2024-08-20T22:16:07.4816157Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:16:07.9128280Z Collecting ninja==1.10.2 2024-08-20T22:16:07.9320554Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:16:07.9408646Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:16:08.4924363Z Installing collected packages: ninja 2024-08-20T22:16:08.5004270Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:16:08.5005869Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:16:08.5050421Z Successfully installed ninja-1.10.2 2024-08-20T22:16:08.5816133Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:08.5818329Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:08.5820275Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:08.5820837Z + install_tlparse 2024-08-20T22:16:08.5821241Z + pip_install --user tlparse==0.3.25 2024-08-20T22:16:08.5821807Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:16:08.9672550Z Collecting tlparse==0.3.25 2024-08-20T22:16:08.9876474Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:16:08.9957602Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:16:09.5695101Z Installing collected packages: tlparse 2024-08-20T22:16:09.6162546Z Successfully installed tlparse-0.3.25 2024-08-20T22:16:09.6934763Z ++ python -m site --user-base 2024-08-20T22:16:09.7175969Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:09.7177798Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:16:09.7178585Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:16:09.7179237Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:09.7180190Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:16:09.7181322Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:16:09.7182224Z + cd test 2024-08-20T22:16:09.7182849Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:16:12.4928123Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:16:12.4928862Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:16:12.4931657Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:16:12.4932454Z + [[ dynamic_aot_eager_torchbench == *pr_time_benchmarks* ]] 2024-08-20T22:16:12.4933295Z + [[ dynamic_aot_eager_torchbench == *dynamo_eager* ]] 2024-08-20T22:16:12.4934012Z + [[ dynamic_aot_eager_torchbench == *aot_eager* ]] 2024-08-20T22:16:12.4934952Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-08-20T22:16:12.4935495Z + [[ dynamic_aot_eager_torchbench == *dynamic* ]] 2024-08-20T22:16:12.4936175Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-08-20T22:16:12.4936770Z + [[ dynamic_aot_eager_torchbench == *cpu* ]] 2024-08-20T22:16:12.4937581Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:16:12.4966554Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:12.4967439Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:16:12.4969281Z + cd test 2024-08-20T22:16:12.4970032Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:16:13.9702478Z PyTorch built with: 2024-08-20T22:16:13.9703047Z - GCC 9.4 2024-08-20T22:16:13.9703486Z - C++ Version: 201703 2024-08-20T22:16:13.9704523Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:13.9705598Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:13.9706276Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:13.9707070Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:16:13.9707664Z - NNPACK is enabled 2024-08-20T22:16:13.9708049Z - CPU capability usage: AVX2 2024-08-20T22:16:13.9708461Z - CUDA Runtime 12.1 2024-08-20T22:16:13.9708985Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:16:13.9709593Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:16:13.9710034Z - Magma 2.6.1 2024-08-20T22:16:13.9716894Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:16:13.9723076Z 2024-08-20T22:16:14.2917987Z + cd test 2024-08-20T22:16:14.2918775Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:16:15.6418308Z ATen/Parallel: 2024-08-20T22:16:15.6418826Z at::get_num_threads() : 8 2024-08-20T22:16:15.6419348Z at::get_num_interop_threads() : 16 2024-08-20T22:16:15.6419894Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:15.6420289Z omp_get_max_threads() : 8 2024-08-20T22:16:15.6421327Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:15.6422112Z mkl_get_max_threads() : 8 2024-08-20T22:16:15.6422757Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:15.6423412Z std::thread::hardware_concurrency() : 16 2024-08-20T22:16:15.6423835Z Environment variables: 2024-08-20T22:16:15.6424178Z OMP_NUM_THREADS : [not set] 2024-08-20T22:16:15.6424540Z MKL_NUM_THREADS : [not set] 2024-08-20T22:16:15.6424923Z ATen parallel backend: OpenMP 2024-08-20T22:16:15.6425173Z 2024-08-20T22:16:15.9200242Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:15.9200875Z + [[ dynamic_aot_eager_torchbench == *backward* ]] 2024-08-20T22:16:15.9201444Z + [[ dynamic_aot_eager_torchbench == *xla* ]] 2024-08-20T22:16:15.9202028Z + [[ dynamic_aot_eager_torchbench == *executorch* ]] 2024-08-20T22:16:15.9202624Z + [[ dynamic_aot_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:16:15.9203362Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:15.9204087Z + [[ dynamic_aot_eager_torchbench == distributed ]] 2024-08-20T22:16:15.9204734Z + [[ dynamic_aot_eager_torchbench == *inductor_distributed* ]] 2024-08-20T22:16:15.9205483Z + [[ dynamic_aot_eager_torchbench == *inductor-halide* ]] 2024-08-20T22:16:15.9206269Z + [[ dynamic_aot_eager_torchbench == *inductor-micro-benchmark* ]] 2024-08-20T22:16:15.9206909Z + [[ dynamic_aot_eager_torchbench == *huggingface* ]] 2024-08-20T22:16:15.9207481Z + [[ dynamic_aot_eager_torchbench == *timm* ]] 2024-08-20T22:16:15.9208059Z + [[ dynamic_aot_eager_torchbench == *torchbench* ]] 2024-08-20T22:16:15.9208578Z + [[ dynamic_aot_eager_torchbench == *cpu* ]] 2024-08-20T22:16:15.9209092Z + install_torchaudio cuda 2024-08-20T22:16:15.9209441Z + local commit 2024-08-20T22:16:15.9210953Z ++ get_pinned_commit audio 2024-08-20T22:16:15.9211416Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T22:16:15.9229771Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:15.9230298Z + [[ cuda == \c\u\d\a ]] 2024-08-20T22:16:15.9230767Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:16:15.9231762Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:15.9233350Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:16.2598404Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:16.2600873Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-gv81xfmp 2024-08-20T22:16:16.2631882Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-gv81xfmp 2024-08-20T22:16:17.1993755Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T22:16:17.2021366Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:17.6091537Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:17.6092664Z Running command git submodule update --init --recursive -q 2024-08-20T22:16:20.0113111Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:16:20.0145038Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T22:16:20.0175363Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T22:16:20.0177573Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T22:16:20.0179845Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T22:16:20.0183489Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T22:16:20.0186899Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T22:16:20.0192358Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1) 2024-08-20T22:16:20.0206839Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T22:16:20.0679883Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T22:16:20.0744847Z Building wheels for collected packages: torchaudio 2024-08-20T22:18:51.6297242Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T22:18:51.6332980Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310373 sha256=f3f8a8b077c0c5c1f8de86669f4cbce5dbf2160f74a503375a4bbad0b6890b18 2024-08-20T22:18:51.6336215Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95 2024-08-20T22:18:51.6363676Z Successfully built torchaudio 2024-08-20T22:18:52.1167857Z Installing collected packages: torchaudio 2024-08-20T22:18:52.3280000Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T22:18:52.5085191Z + install_torchtext 2024-08-20T22:18:52.5085687Z + local data_commit 2024-08-20T22:18:52.5086095Z + local text_commit 2024-08-20T22:18:52.5089047Z ++ get_pinned_commit data 2024-08-20T22:18:52.5089466Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T22:18:52.5109914Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.5113295Z ++ get_pinned_commit text 2024-08-20T22:18:52.5114832Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T22:18:52.5130971Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:18:52.5132277Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.5133717Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.8893833Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.8898213Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-47tveix1 2024-08-20T22:18:52.8940337Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-47tveix1 2024-08-20T22:18:53.3267892Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T22:18:53.3296140Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:53.5936030Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:53.7432639Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:53.7433891Z Running command git submodule update --init --recursive -q 2024-08-20T22:20:13.5410130Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:20:13.5447033Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T22:20:13.5449408Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T22:20:13.5454088Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T22:20:13.5483953Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T22:20:13.5488452Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T22:20:13.5492051Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T22:20:13.5495400Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T22:20:13.5498334Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T22:20:13.5503561Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1) 2024-08-20T22:20:13.5518177Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T22:20:13.5579149Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T22:20:13.5583438Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T22:20:13.5587022Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T22:20:13.6084018Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T22:20:13.6147017Z Building wheels for collected packages: torchdata 2024-08-20T22:20:15.5883676Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T22:20:15.5891536Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=4448e3dc5f15831138b935a862c03940b629ba584b743e8c6ae801dfa1263322 2024-08-20T22:20:15.5893425Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-08-20T22:20:15.5920847Z Successfully built torchdata 2024-08-20T22:20:16.0784327Z Installing collected packages: torchdata 2024-08-20T22:20:16.1740576Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T22:20:21.1909007Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:21.1910669Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:21.5276428Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:21.5279696Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-stwkvynf 2024-08-20T22:20:21.5310936Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-stwkvynf 2024-08-20T22:20:22.3323451Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T22:20:22.3354106Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:22.8239771Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:23.0224621Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:23.0225896Z Running command git submodule update --init --recursive -q 2024-08-20T22:20:30.5506056Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:20:30.5540151Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T22:20:30.5543362Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T22:20:30.5570881Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T22:20:30.5572591Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4) 2024-08-20T22:20:30.5574588Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T22:20:30.5578421Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T22:20:30.5581692Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T22:20:30.5585498Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T22:20:30.5588859Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T22:20:30.5615185Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T22:20:30.5618837Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T22:20:30.5621319Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T22:20:30.5624704Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T22:20:30.5627732Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T22:20:30.5632716Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1) 2024-08-20T22:20:30.5647467Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T22:20:30.6227223Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T22:20:30.6293682Z Building wheels for collected packages: torchtext 2024-08-20T22:21:08.5145360Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:21:08.5179099Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045463 sha256=e0c83c063a1706815e836ddca2e37ba1c2210efedb5ae3f21ca7277889400342 2024-08-20T22:21:08.5181595Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-08-20T22:21:08.5203054Z Successfully built torchtext 2024-08-20T22:21:08.9989853Z Installing collected packages: torchtext 2024-08-20T22:21:09.1115211Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T22:21:09.2813816Z + install_torchvision 2024-08-20T22:21:09.2814274Z + local orig_preload 2024-08-20T22:21:09.2814599Z + local commit 2024-08-20T22:21:09.2817376Z ++ get_pinned_commit vision 2024-08-20T22:21:09.2817904Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:09.2837625Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:09.2838230Z + orig_preload= 2024-08-20T22:21:09.2838839Z + '[' -n '' ']' 2024-08-20T22:21:09.2842322Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:09.2843801Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:09.6210908Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:09.6215816Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-pq0opoyp 2024-08-20T22:21:09.6245849Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-pq0opoyp 2024-08-20T22:21:11.1214842Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:11.1244273Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.4641986Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.7680078Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:15.2547735Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:15.2580595Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:15.2584997Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:15.2589667Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:15.2660755Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:15.2666501Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:15.2669963Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:15.2673322Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:15.2676289Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:15.2683244Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:15.2698777Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:15.3177559Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:15.3240960Z Building wheels for collected packages: torchvision 2024-08-20T22:22:32.3987283Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:22:32.4020205Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2115919 sha256=1eeb97d82aeb0585683dfbc9c2bb8a801d07730e8ead229e62a92d5d8809ac42 2024-08-20T22:22:32.4021928Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:32.4059214Z Successfully built torchvision 2024-08-20T22:22:32.8922053Z Installing collected packages: torchvision 2024-08-20T22:22:33.3043665Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:33.4512261Z + '[' -n '' ']' 2024-08-20T22:22:33.4512840Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:22:33.4513346Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T22:22:33.4515778Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T22:22:33.7881514Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T22:22:33.7886158Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-wzz03u3j 2024-08-20T22:22:33.7916820Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-wzz03u3j 2024-08-20T22:22:34.3668953Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:22:36.8074617Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:22:36.8108140Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T22:23:55.1199988Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / done 2024-08-20T22:23:55.1217795Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928013 sha256=f4549293e100cc7c02b830e0a40fd1ff62227ff74ef41e2f0570ff413fe9b71c 2024-08-20T22:23:55.1220986Z Stored in directory: /tmp/pip-ephem-wheel-cache-5m7c11xn/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2024-08-20T22:23:55.1250948Z Successfully built torchao 2024-08-20T22:23:55.6767642Z Installing collected packages: torchao 2024-08-20T22:23:55.9192519Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T22:23:56.0315439Z + id=1 2024-08-20T22:23:56.0317815Z + pip_install opencv-python==4.8.0.74 2024-08-20T22:23:56.0318442Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T22:23:56.5032212Z Collecting opencv-python==4.8.0.74 2024-08-20T22:23:56.5239462Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T22:23:56.5323226Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2024-08-20T22:23:56.5391633Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T22:23:57.5516737Z Installing collected packages: opencv-python 2024-08-20T22:23:58.4243882Z Successfully installed opencv-python-4.8.0.74 2024-08-20T22:23:58.5273848Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T22:23:58.5274649Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T22:23:58.5277431Z + [[ dynamic_aot_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-08-20T22:23:58.5278009Z + checkout_install_torchbench 2024-08-20T22:23:58.5278384Z + local commit 2024-08-20T22:23:58.5278708Z ++ get_pinned_commit torchbench 2024-08-20T22:23:58.5279138Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T22:23:58.5295651Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:23:58.5296258Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T22:23:58.5313374Z Cloning into 'torchbench'... 2024-08-20T22:23:58.6877355Z remote: Enumerating objects: 31001, done. 2024-08-20T22:23:58.6879953Z remote: Counting objects: 0% (1/1741) 2024-08-20T22:23:58.6882383Z remote: Counting objects: 1% (18/1741) 2024-08-20T22:23:58.6884294Z remote: Counting objects: 2% (35/1741) 2024-08-20T22:23:58.6885007Z remote: Counting objects: 3% (53/1741) 2024-08-20T22:23:58.6885743Z remote: Counting objects: 4% (70/1741) 2024-08-20T22:23:58.6887357Z remote: Counting objects: 5% (88/1741) 2024-08-20T22:23:58.6888132Z remote: Counting objects: 6% (105/1741) 2024-08-20T22:23:58.6888704Z remote: Counting objects: 7% (122/1741) 2024-08-20T22:23:58.6889472Z remote: Counting objects: 8% (140/1741) 2024-08-20T22:23:58.6890402Z remote: Counting objects: 9% (157/1741) 2024-08-20T22:23:58.6892226Z remote: Counting objects: 10% (175/1741) 2024-08-20T22:23:58.6892969Z remote: Counting objects: 11% (192/1741) 2024-08-20T22:23:58.6893686Z remote: Counting objects: 12% (209/1741) 2024-08-20T22:23:58.6894243Z remote: Counting objects: 13% (227/1741) 2024-08-20T22:23:58.6895156Z remote: Counting objects: 14% (244/1741) 2024-08-20T22:23:58.6895960Z remote: Counting objects: 15% (262/1741) 2024-08-20T22:23:58.6896627Z remote: Counting objects: 16% (279/1741) 2024-08-20T22:23:58.6897155Z remote: Counting objects: 17% (296/1741) 2024-08-20T22:23:58.6897703Z remote: Counting objects: 18% (314/1741) 2024-08-20T22:23:58.6898221Z remote: Counting objects: 19% (331/1741) 2024-08-20T22:23:58.6899180Z remote: Counting objects: 20% (349/1741) 2024-08-20T22:23:58.6899697Z remote: Counting objects: 21% (366/1741) 2024-08-20T22:23:58.6900210Z remote: Counting objects: 22% (384/1741) 2024-08-20T22:23:58.6900773Z remote: Counting objects: 23% (401/1741) 2024-08-20T22:23:58.6901284Z remote: Counting objects: 24% (418/1741) 2024-08-20T22:23:58.6901805Z remote: Counting objects: 25% (436/1741) 2024-08-20T22:23:58.6902322Z remote: Counting objects: 26% (453/1741) 2024-08-20T22:23:58.6902834Z remote: Counting objects: 27% (471/1741) 2024-08-20T22:23:58.6903343Z remote: Counting objects: 28% (488/1741) 2024-08-20T22:23:58.6904054Z remote: Counting objects: 29% (505/1741) 2024-08-20T22:23:58.6904574Z remote: Counting objects: 30% (523/1741) 2024-08-20T22:23:58.6905082Z remote: Counting objects: 31% (540/1741) 2024-08-20T22:23:58.6905594Z remote: Counting objects: 32% (558/1741) 2024-08-20T22:23:58.6906112Z remote: Counting objects: 33% (575/1741) 2024-08-20T22:23:58.6906624Z remote: Counting objects: 34% (592/1741) 2024-08-20T22:23:58.6907129Z remote: Counting objects: 35% (610/1741) 2024-08-20T22:23:58.6907775Z remote: Counting objects: 36% (627/1741) 2024-08-20T22:23:58.6908291Z remote: Counting objects: 37% (645/1741) 2024-08-20T22:23:58.6908803Z remote: Counting objects: 38% (662/1741) 2024-08-20T22:23:58.6909312Z remote: Counting objects: 39% (679/1741) 2024-08-20T22:23:58.6909822Z remote: Counting objects: 40% (697/1741) 2024-08-20T22:23:58.6910335Z remote: Counting objects: 41% (714/1741) 2024-08-20T22:23:58.6910898Z remote: Counting objects: 42% (732/1741) 2024-08-20T22:23:58.6911413Z remote: Counting objects: 43% (749/1741) 2024-08-20T22:23:58.6911924Z remote: Counting objects: 44% (767/1741) 2024-08-20T22:23:58.6912437Z remote: Counting objects: 45% (784/1741) 2024-08-20T22:23:58.6912956Z remote: Counting objects: 46% (801/1741) 2024-08-20T22:23:58.6913467Z remote: Counting objects: 47% (819/1741) 2024-08-20T22:23:58.6913979Z remote: Counting objects: 48% (836/1741) 2024-08-20T22:23:58.6914492Z remote: Counting objects: 49% (854/1741) 2024-08-20T22:23:58.6914995Z remote: Counting objects: 50% (871/1741) 2024-08-20T22:23:58.6915508Z remote: Counting objects: 51% (888/1741) 2024-08-20T22:23:58.6916020Z remote: Counting objects: 52% (906/1741) 2024-08-20T22:23:58.6916531Z remote: Counting objects: 53% (923/1741) 2024-08-20T22:23:58.6917037Z remote: Counting objects: 54% (941/1741) 2024-08-20T22:23:58.6917549Z remote: Counting objects: 55% (958/1741) 2024-08-20T22:23:58.6918060Z remote: Counting objects: 56% (975/1741) 2024-08-20T22:23:58.6918579Z remote: Counting objects: 57% (993/1741) 2024-08-20T22:23:58.6919091Z remote: Counting objects: 58% (1010/1741) 2024-08-20T22:23:58.6919618Z remote: Counting objects: 59% (1028/1741) 2024-08-20T22:23:58.6920147Z remote: Counting objects: 60% (1045/1741) 2024-08-20T22:23:58.6920677Z remote: Counting objects: 61% (1063/1741) 2024-08-20T22:23:58.6921194Z remote: Counting objects: 62% (1080/1741) 2024-08-20T22:23:58.6921716Z remote: Counting objects: 63% (1097/1741) 2024-08-20T22:23:58.6922236Z remote: Counting objects: 64% (1115/1741) 2024-08-20T22:23:58.6922758Z remote: Counting objects: 65% (1132/1741) 2024-08-20T22:23:58.6923272Z remote: Counting objects: 66% (1150/1741) 2024-08-20T22:23:58.6923794Z remote: Counting objects: 67% (1167/1741) 2024-08-20T22:23:58.6924313Z remote: Counting objects: 68% (1184/1741) 2024-08-20T22:23:58.6924824Z remote: Counting objects: 69% (1202/1741) 2024-08-20T22:23:58.6925348Z remote: Counting objects: 70% (1219/1741) 2024-08-20T22:23:58.6925867Z remote: Counting objects: 71% (1237/1741) 2024-08-20T22:23:58.6926389Z remote: Counting objects: 72% (1254/1741) 2024-08-20T22:23:58.6926899Z remote: Counting objects: 73% (1271/1741) 2024-08-20T22:23:58.6927415Z remote: Counting objects: 74% (1289/1741) 2024-08-20T22:23:58.6928034Z remote: Counting objects: 75% (1306/1741) 2024-08-20T22:23:58.6928546Z remote: Counting objects: 76% (1324/1741) 2024-08-20T22:23:58.6929058Z remote: Counting objects: 77% (1341/1741) 2024-08-20T22:23:58.6929570Z remote: Counting objects: 78% (1358/1741) 2024-08-20T22:23:58.6930085Z remote: Counting objects: 79% (1376/1741) 2024-08-20T22:23:58.6930632Z remote: Counting objects: 80% (1393/1741) 2024-08-20T22:23:58.6931160Z remote: Counting objects: 81% (1411/1741) 2024-08-20T22:23:58.6931673Z remote: Counting objects: 82% (1428/1741) 2024-08-20T22:23:58.6932187Z remote: Counting objects: 83% (1446/1741) 2024-08-20T22:23:58.6932786Z remote: Counting objects: 84% (1463/1741) 2024-08-20T22:23:58.6933299Z remote: Counting objects: 85% (1480/1741) 2024-08-20T22:23:58.6933817Z remote: Counting objects: 86% (1498/1741) 2024-08-20T22:23:58.6934331Z remote: Counting objects: 87% (1515/1741) 2024-08-20T22:23:58.6934852Z remote: Counting objects: 88% (1533/1741) 2024-08-20T22:23:58.6935360Z remote: Counting objects: 89% (1550/1741) 2024-08-20T22:23:58.6935879Z remote: Counting objects: 90% (1567/1741) 2024-08-20T22:23:58.6936393Z remote: Counting objects: 91% (1585/1741) 2024-08-20T22:23:58.6936901Z remote: Counting objects: 92% (1602/1741) 2024-08-20T22:23:58.6937823Z remote: Counting objects: 93% (1620/1741) 2024-08-20T22:23:58.6938346Z remote: Counting objects: 94% (1637/1741) 2024-08-20T22:23:58.6938866Z remote: Counting objects: 95% (1654/1741) 2024-08-20T22:23:58.6939377Z remote: Counting objects: 96% (1672/1741) 2024-08-20T22:23:58.6939899Z remote: Counting objects: 97% (1689/1741) 2024-08-20T22:23:58.6940420Z remote: Counting objects: 98% (1707/1741) 2024-08-20T22:23:58.6940986Z remote: Counting objects: 99% (1724/1741) 2024-08-20T22:23:58.6941501Z remote: Counting objects: 100% (1741/1741) 2024-08-20T22:23:58.6942055Z remote: Counting objects: 100% (1741/1741), done. 2024-08-20T22:23:58.6942627Z remote: Compressing objects: 0% (1/897) 2024-08-20T22:23:58.6984444Z remote: Compressing objects: 1% (9/897) 2024-08-20T22:23:58.7101729Z remote: Compressing objects: 2% (18/897) 2024-08-20T22:23:58.7187305Z remote: Compressing objects: 3% (27/897) 2024-08-20T22:23:58.7236069Z remote: Compressing objects: 4% (36/897) 2024-08-20T22:23:58.7287180Z remote: Compressing objects: 5% (45/897) 2024-08-20T22:23:58.7321464Z remote: Compressing objects: 6% (54/897) 2024-08-20T22:23:58.7349959Z remote: Compressing objects: 7% (63/897) 2024-08-20T22:23:58.7384152Z remote: Compressing objects: 8% (72/897) 2024-08-20T22:23:58.7421285Z remote: Compressing objects: 9% (81/897) 2024-08-20T22:23:58.7445878Z remote: Compressing objects: 10% (90/897) 2024-08-20T22:23:58.7467435Z remote: Compressing objects: 11% (99/897) 2024-08-20T22:23:58.7474077Z remote: Compressing objects: 12% (108/897) 2024-08-20T22:23:58.7489174Z remote: Compressing objects: 13% (117/897) 2024-08-20T22:23:58.7504125Z remote: Compressing objects: 14% (126/897) 2024-08-20T22:23:58.7514163Z remote: Compressing objects: 15% (135/897) 2024-08-20T22:23:58.7522142Z remote: Compressing objects: 16% (144/897) 2024-08-20T22:23:58.7529515Z remote: Compressing objects: 17% (153/897) 2024-08-20T22:23:58.7532962Z remote: Compressing objects: 18% (162/897) 2024-08-20T22:23:58.7538279Z remote: Compressing objects: 19% (171/897) 2024-08-20T22:23:58.7538905Z remote: Compressing objects: 20% (180/897) 2024-08-20T22:23:58.7541060Z remote: Compressing objects: 21% (189/897) 2024-08-20T22:23:58.7542042Z remote: Compressing objects: 22% (198/897) 2024-08-20T22:23:58.7544424Z remote: Compressing objects: 23% (207/897) 2024-08-20T22:23:58.7545207Z remote: Compressing objects: 24% (216/897) 2024-08-20T22:23:58.7545908Z remote: Compressing objects: 25% (225/897) 2024-08-20T22:23:58.7548123Z remote: Compressing objects: 26% (234/897) 2024-08-20T22:23:58.7554886Z remote: Compressing objects: 27% (243/897) 2024-08-20T22:23:58.7557154Z remote: Compressing objects: 28% (252/897) 2024-08-20T22:23:58.7560570Z remote: Compressing objects: 29% (261/897) 2024-08-20T22:23:58.7563089Z remote: Compressing objects: 30% (270/897) 2024-08-20T22:23:58.7566056Z remote: Compressing objects: 31% (279/897) 2024-08-20T22:23:58.7568931Z remote: Compressing objects: 32% (288/897) 2024-08-20T22:23:58.7572165Z remote: Compressing objects: 33% (297/897) 2024-08-20T22:23:58.7574594Z remote: Compressing objects: 34% (305/897) 2024-08-20T22:23:58.7577865Z remote: Compressing objects: 35% (314/897) 2024-08-20T22:23:58.7580588Z remote: Compressing objects: 36% (323/897) 2024-08-20T22:23:58.7583659Z remote: Compressing objects: 37% (332/897) 2024-08-20T22:23:58.7584335Z remote: Compressing objects: 38% (341/897) 2024-08-20T22:23:58.7587113Z remote: Compressing objects: 39% (350/897) 2024-08-20T22:23:58.7590152Z remote: Compressing objects: 40% (359/897) 2024-08-20T22:23:58.7590976Z remote: Compressing objects: 41% (368/897) 2024-08-20T22:23:58.7591767Z remote: Compressing objects: 42% (377/897) 2024-08-20T22:23:58.7592578Z remote: Compressing objects: 43% (386/897) 2024-08-20T22:23:58.7593335Z remote: Compressing objects: 44% (395/897) 2024-08-20T22:23:58.7594031Z remote: Compressing objects: 45% (404/897) 2024-08-20T22:23:58.7594769Z remote: Compressing objects: 46% (413/897) 2024-08-20T22:23:58.7595664Z remote: Compressing objects: 47% (422/897) 2024-08-20T22:23:58.7596295Z remote: Compressing objects: 48% (431/897) 2024-08-20T22:23:58.7596992Z remote: Compressing objects: 49% (440/897) 2024-08-20T22:23:58.7597553Z remote: Compressing objects: 50% (449/897) 2024-08-20T22:23:58.7598176Z remote: Compressing objects: 51% (458/897) 2024-08-20T22:23:58.7598907Z remote: Compressing objects: 52% (467/897) 2024-08-20T22:23:58.7599573Z remote: Compressing objects: 53% (476/897) 2024-08-20T22:23:58.7600208Z remote: Compressing objects: 54% (485/897) 2024-08-20T22:23:58.7600789Z remote: Compressing objects: 55% (494/897) 2024-08-20T22:23:58.7601337Z remote: Compressing objects: 56% (503/897) 2024-08-20T22:23:58.7601972Z remote: Compressing objects: 57% (512/897) 2024-08-20T22:23:58.7602719Z remote: Compressing objects: 58% (521/897) 2024-08-20T22:23:58.7603259Z remote: Compressing objects: 59% (530/897) 2024-08-20T22:23:58.7603798Z remote: Compressing objects: 60% (539/897) 2024-08-20T22:23:58.7604329Z remote: Compressing objects: 61% (548/897) 2024-08-20T22:23:58.7604862Z remote: Compressing objects: 62% (557/897) 2024-08-20T22:23:58.7605406Z remote: Compressing objects: 63% (566/897) 2024-08-20T22:23:58.7605932Z remote: Compressing objects: 64% (575/897) 2024-08-20T22:23:58.7606472Z remote: Compressing objects: 65% (584/897) 2024-08-20T22:23:58.7607001Z remote: Compressing objects: 66% (593/897) 2024-08-20T22:23:58.7607543Z remote: Compressing objects: 67% (601/897) 2024-08-20T22:23:58.7608066Z remote: Compressing objects: 68% (610/897) 2024-08-20T22:23:58.7608602Z remote: Compressing objects: 69% (619/897) 2024-08-20T22:23:58.7609133Z remote: Compressing objects: 70% (628/897) 2024-08-20T22:23:58.7609676Z remote: Compressing objects: 71% (637/897) 2024-08-20T22:23:58.7610202Z remote: Compressing objects: 72% (646/897) 2024-08-20T22:23:58.7610860Z remote: Compressing objects: 73% (655/897) 2024-08-20T22:23:58.7611489Z remote: Compressing objects: 74% (664/897) 2024-08-20T22:23:58.7612037Z remote: Compressing objects: 75% (673/897) 2024-08-20T22:23:58.7613228Z remote: Compressing objects: 76% (682/897) 2024-08-20T22:23:58.7621608Z remote: Compressing objects: 77% (691/897) 2024-08-20T22:23:58.7625774Z remote: Compressing objects: 78% (700/897) 2024-08-20T22:23:58.7632224Z remote: Compressing objects: 79% (709/897) 2024-08-20T22:23:58.7638586Z remote: Compressing objects: 80% (718/897) 2024-08-20T22:23:58.7647194Z remote: Compressing objects: 81% (727/897) 2024-08-20T22:23:58.7654245Z remote: Compressing objects: 82% (736/897) 2024-08-20T22:23:58.7657815Z remote: Compressing objects: 83% (745/897) 2024-08-20T22:23:58.7661571Z remote: Compressing objects: 84% (754/897) 2024-08-20T22:23:58.7665384Z remote: Compressing objects: 85% (763/897) 2024-08-20T22:23:58.7670302Z remote: Compressing objects: 86% (772/897) 2024-08-20T22:23:58.7674514Z remote: Compressing objects: 87% (781/897) 2024-08-20T22:23:58.7679671Z remote: Compressing objects: 88% (790/897) 2024-08-20T22:23:58.7682385Z remote: Compressing objects: 89% (799/897) 2024-08-20T22:23:58.7685582Z remote: Compressing objects: 90% (808/897) 2024-08-20T22:23:58.7688333Z remote: Compressing objects: 91% (817/897) 2024-08-20T22:23:58.7691335Z remote: Compressing objects: 92% (826/897) 2024-08-20T22:23:58.7695080Z remote: Compressing objects: 93% (835/897) 2024-08-20T22:23:58.7698842Z remote: Compressing objects: 94% (844/897) 2024-08-20T22:23:58.7701608Z remote: Compressing objects: 95% (853/897) 2024-08-20T22:23:58.7704773Z remote: Compressing objects: 96% (862/897) 2024-08-20T22:23:58.7707728Z remote: Compressing objects: 97% (871/897) 2024-08-20T22:23:58.7708440Z remote: Compressing objects: 98% (880/897) 2024-08-20T22:23:58.7709241Z remote: Compressing objects: 99% (889/897) 2024-08-20T22:23:58.7709944Z remote: Compressing objects: 100% (897/897) 2024-08-20T22:23:58.7710549Z remote: Compressing objects: 100% (897/897), done. 2024-08-20T22:23:58.7796177Z Receiving objects: 0% (1/31001) 2024-08-20T22:23:58.7853900Z Receiving objects: 1% (311/31001) 2024-08-20T22:23:58.7908517Z Receiving objects: 2% (621/31001) 2024-08-20T22:23:58.7954376Z Receiving objects: 3% (931/31001) 2024-08-20T22:23:58.7987162Z Receiving objects: 4% (1241/31001) 2024-08-20T22:23:58.8019233Z Receiving objects: 5% (1551/31001) 2024-08-20T22:23:58.8057176Z Receiving objects: 6% (1861/31001) 2024-08-20T22:23:58.8095936Z Receiving objects: 7% (2171/31001) 2024-08-20T22:23:58.8125914Z Receiving objects: 8% (2481/31001) 2024-08-20T22:23:58.8158827Z Receiving objects: 9% (2791/31001) 2024-08-20T22:23:58.8228165Z Receiving objects: 10% (3101/31001) 2024-08-20T22:23:58.8276394Z Receiving objects: 11% (3411/31001) 2024-08-20T22:23:58.8342225Z Receiving objects: 12% (3721/31001) 2024-08-20T22:23:58.8472835Z Receiving objects: 13% (4031/31001) 2024-08-20T22:23:58.8617105Z Receiving objects: 14% (4341/31001) 2024-08-20T22:23:58.8684731Z Receiving objects: 15% (4651/31001) 2024-08-20T22:23:58.8760281Z Receiving objects: 16% (4961/31001) 2024-08-20T22:23:58.8815587Z Receiving objects: 17% (5271/31001) 2024-08-20T22:23:58.8847145Z Receiving objects: 18% (5581/31001) 2024-08-20T22:23:58.8903858Z Receiving objects: 19% (5891/31001) 2024-08-20T22:23:58.8938145Z Receiving objects: 20% (6201/31001) 2024-08-20T22:23:58.8953142Z Receiving objects: 21% (6511/31001) 2024-08-20T22:23:58.8998197Z Receiving objects: 22% (6821/31001) 2024-08-20T22:23:59.7734897Z Receiving objects: 23% (7131/31001) 2024-08-20T22:23:59.8791111Z Receiving objects: 23% (7393/31001), 75.39 MiB | 75.39 MiB/s 2024-08-20T22:24:00.4403574Z Receiving objects: 24% (7441/31001), 75.39 MiB | 75.39 MiB/s 2024-08-20T22:24:00.5127774Z Receiving objects: 25% (7751/31001), 113.12 MiB | 75.46 MiB/s 2024-08-20T22:24:00.5846280Z Receiving objects: 26% (8061/31001), 113.12 MiB | 75.46 MiB/s 2024-08-20T22:24:00.6561597Z Receiving objects: 27% (8371/31001), 113.12 MiB | 75.46 MiB/s 2024-08-20T22:24:00.7270725Z Receiving objects: 28% (8681/31001), 113.12 MiB | 75.46 MiB/s 2024-08-20T22:24:00.7733250Z Receiving objects: 29% (8991/31001), 113.12 MiB | 75.46 MiB/s 2024-08-20T22:24:00.7989911Z Receiving objects: 29% (9192/31001), 113.12 MiB | 75.46 MiB/s 2024-08-20T22:24:00.8714617Z Receiving objects: 30% (9301/31001), 151.43 MiB | 75.75 MiB/s 2024-08-20T22:24:00.9427151Z Receiving objects: 31% (9611/31001), 151.43 MiB | 75.75 MiB/s 2024-08-20T22:24:01.1784764Z Receiving objects: 32% (9921/31001), 151.43 MiB | 75.75 MiB/s 2024-08-20T22:24:01.2530279Z Receiving objects: 33% (10231/31001), 151.43 MiB | 75.75 MiB/s 2024-08-20T22:24:01.2654956Z Receiving objects: 34% (10541/31001), 151.43 MiB | 75.75 MiB/s 2024-08-20T22:24:01.5252618Z Receiving objects: 35% (10851/31001), 151.43 MiB | 75.75 MiB/s 2024-08-20T22:24:01.7729974Z Receiving objects: 36% (11161/31001), 184.87 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7733372Z Receiving objects: 37% (11471/31001), 184.87 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7748676Z Receiving objects: 37% (11522/31001), 184.87 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7771801Z Receiving objects: 38% (11781/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7789333Z Receiving objects: 39% (12091/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7803836Z Receiving objects: 40% (12401/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7814427Z Receiving objects: 41% (12711/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7826636Z Receiving objects: 42% (13021/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7842606Z Receiving objects: 43% (13331/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7858483Z Receiving objects: 44% (13641/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7866112Z Receiving objects: 45% (13951/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7871265Z Receiving objects: 46% (14261/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7884987Z Receiving objects: 47% (14571/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7905701Z Receiving objects: 48% (14881/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.7918538Z Receiving objects: 49% (15191/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8117517Z Receiving objects: 50% (15501/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8122956Z Receiving objects: 51% (15811/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8129826Z Receiving objects: 52% (16121/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8136096Z Receiving objects: 53% (16431/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8141736Z Receiving objects: 54% (16741/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8158438Z Receiving objects: 55% (17051/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8168682Z Receiving objects: 56% (17361/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8174847Z Receiving objects: 57% (17671/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8188342Z Receiving objects: 58% (17981/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8199920Z Receiving objects: 59% (18291/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8209980Z Receiving objects: 60% (18601/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8346963Z Receiving objects: 61% (18911/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8352966Z Receiving objects: 62% (19221/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8358543Z Receiving objects: 63% (19531/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8366363Z Receiving objects: 64% (19841/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8373789Z Receiving objects: 65% (20151/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8382373Z Receiving objects: 66% (20461/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.8637992Z Receiving objects: 67% (20771/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9034454Z Receiving objects: 68% (21081/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9041769Z Receiving objects: 69% (21391/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9054376Z Receiving objects: 70% (21701/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9065182Z Receiving objects: 71% (22011/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9079919Z Receiving objects: 72% (22321/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9088927Z Receiving objects: 73% (22631/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9104216Z Receiving objects: 74% (22941/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9115022Z Receiving objects: 75% (23251/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9122192Z Receiving objects: 76% (23561/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9130020Z Receiving objects: 77% (23871/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9341904Z Receiving objects: 78% (24181/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9375864Z Receiving objects: 79% (24491/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9384530Z Receiving objects: 80% (24801/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9396715Z Receiving objects: 81% (25111/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9408955Z Receiving objects: 82% (25421/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9456832Z Receiving objects: 83% (25731/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9594272Z Receiving objects: 84% (26041/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9599756Z Receiving objects: 85% (26351/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9607338Z Receiving objects: 86% (26661/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9612497Z Receiving objects: 87% (26971/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9619878Z Receiving objects: 88% (27281/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9625806Z Receiving objects: 89% (27591/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9736600Z Receiving objects: 90% (27901/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9750792Z Receiving objects: 91% (28211/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9762559Z Receiving objects: 92% (28521/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9779300Z Receiving objects: 93% (28831/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9812608Z Receiving objects: 94% (29141/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9824460Z Receiving objects: 95% (29451/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9830127Z Receiving objects: 96% (29761/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9837614Z Receiving objects: 97% (30071/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9911633Z Receiving objects: 98% (30381/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9931232Z Receiving objects: 99% (30691/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9932971Z remote: Total 31001 (delta 1016), reused 1474 (delta 825), pack-reused 29260 (from 1) 2024-08-20T22:24:01.9948899Z Receiving objects: 100% (31001/31001), 221.84 MiB | 73.97 MiB/s 2024-08-20T22:24:01.9949584Z Receiving objects: 100% (31001/31001), 232.50 MiB | 72.19 MiB/s, done. 2024-08-20T22:24:01.9980969Z Resolving deltas: 0% (0/16729) 2024-08-20T22:24:01.9994827Z Resolving deltas: 1% (168/16729) 2024-08-20T22:24:02.0004650Z Resolving deltas: 2% (402/16729) 2024-08-20T22:24:02.0028114Z Resolving deltas: 3% (512/16729) 2024-08-20T22:24:02.0044630Z Resolving deltas: 4% (673/16729) 2024-08-20T22:24:02.0053567Z Resolving deltas: 5% (902/16729) 2024-08-20T22:24:02.0063404Z Resolving deltas: 6% (1018/16729) 2024-08-20T22:24:02.0076454Z Resolving deltas: 7% (1172/16729) 2024-08-20T22:24:02.0085402Z Resolving deltas: 8% (1359/16729) 2024-08-20T22:24:02.0096928Z Resolving deltas: 9% (1518/16729) 2024-08-20T22:24:02.0107893Z Resolving deltas: 10% (1676/16729) 2024-08-20T22:24:02.0120844Z Resolving deltas: 11% (1843/16729) 2024-08-20T22:24:02.0134041Z Resolving deltas: 12% (2008/16729) 2024-08-20T22:24:02.0148862Z Resolving deltas: 13% (2176/16729) 2024-08-20T22:24:02.0161675Z Resolving deltas: 14% (2401/16729) 2024-08-20T22:24:02.0177486Z Resolving deltas: 15% (2511/16729) 2024-08-20T22:24:02.0200877Z Resolving deltas: 16% (2681/16729) 2024-08-20T22:24:02.0209064Z Resolving deltas: 18% (3013/16729) 2024-08-20T22:24:02.0217057Z Resolving deltas: 19% (3186/16729) 2024-08-20T22:24:02.0257686Z Resolving deltas: 20% (3357/16729) 2024-08-20T22:24:02.0285588Z Resolving deltas: 21% (3516/16729) 2024-08-20T22:24:02.0286805Z Resolving deltas: 22% (3842/16729) 2024-08-20T22:24:02.0301070Z Resolving deltas: 23% (3864/16729) 2024-08-20T22:24:02.0318399Z Resolving deltas: 24% (4015/16729) 2024-08-20T22:24:02.0337067Z Resolving deltas: 25% (4206/16729) 2024-08-20T22:24:02.0345963Z Resolving deltas: 26% (4444/16729) 2024-08-20T22:24:02.0383989Z Resolving deltas: 27% (4543/16729) 2024-08-20T22:24:02.0403457Z Resolving deltas: 28% (4687/16729) 2024-08-20T22:24:02.0418967Z Resolving deltas: 29% (4855/16729) 2024-08-20T22:24:02.0438576Z Resolving deltas: 30% (5019/16729) 2024-08-20T22:24:02.0453814Z Resolving deltas: 31% (5201/16729) 2024-08-20T22:24:02.0467102Z Resolving deltas: 32% (5364/16729) 2024-08-20T22:24:02.0480011Z Resolving deltas: 33% (5521/16729) 2024-08-20T22:24:02.0488830Z Resolving deltas: 34% (5725/16729) 2024-08-20T22:24:02.0503771Z Resolving deltas: 35% (5857/16729) 2024-08-20T22:24:02.0524623Z Resolving deltas: 36% (6027/16729) 2024-08-20T22:24:02.0542568Z Resolving deltas: 37% (6194/16729) 2024-08-20T22:24:02.0557221Z Resolving deltas: 38% (6361/16729) 2024-08-20T22:24:02.0568337Z Resolving deltas: 39% (6557/16729) 2024-08-20T22:24:02.0589504Z Resolving deltas: 40% (6693/16729) 2024-08-20T22:24:02.0601867Z Resolving deltas: 41% (6885/16729) 2024-08-20T22:24:02.0619398Z Resolving deltas: 42% (7036/16729) 2024-08-20T22:24:02.0632754Z Resolving deltas: 43% (7214/16729) 2024-08-20T22:24:02.0642681Z Resolving deltas: 44% (7363/16729) 2024-08-20T22:24:02.0653876Z Resolving deltas: 45% (7548/16729) 2024-08-20T22:24:02.0670135Z Resolving deltas: 46% (7702/16729) 2024-08-20T22:24:02.0694340Z Resolving deltas: 47% (7866/16729) 2024-08-20T22:24:02.0703192Z Resolving deltas: 48% (8030/16729) 2024-08-20T22:24:02.0722085Z Resolving deltas: 49% (8198/16729) 2024-08-20T22:24:02.0731557Z Resolving deltas: 50% (8366/16729) 2024-08-20T22:24:02.0740080Z Resolving deltas: 51% (8532/16729) 2024-08-20T22:24:02.0750949Z Resolving deltas: 52% (8703/16729) 2024-08-20T22:24:02.0765077Z Resolving deltas: 53% (8879/16729) 2024-08-20T22:24:02.0777101Z Resolving deltas: 54% (9038/16729) 2024-08-20T22:24:02.0793275Z Resolving deltas: 55% (9202/16729) 2024-08-20T22:24:02.0801106Z Resolving deltas: 56% (9373/16729) 2024-08-20T22:24:02.0811594Z Resolving deltas: 57% (9567/16729) 2024-08-20T22:24:02.0818953Z Resolving deltas: 58% (9814/16729) 2024-08-20T22:24:02.0828016Z Resolving deltas: 59% (9924/16729) 2024-08-20T22:24:02.0835357Z Resolving deltas: 60% (10094/16729) 2024-08-20T22:24:02.0854659Z Resolving deltas: 61% (10212/16729) 2024-08-20T22:24:02.0858750Z Resolving deltas: 62% (10459/16729) 2024-08-20T22:24:02.0896967Z Resolving deltas: 63% (10540/16729) 2024-08-20T22:24:02.0912150Z Resolving deltas: 65% (10972/16729) 2024-08-20T22:24:02.0916394Z Resolving deltas: 66% (11186/16729) 2024-08-20T22:24:02.0935804Z Resolving deltas: 67% (11255/16729) 2024-08-20T22:24:02.0945169Z Resolving deltas: 68% (11376/16729) 2024-08-20T22:24:02.0956822Z Resolving deltas: 69% (11567/16729) 2024-08-20T22:24:02.0973412Z Resolving deltas: 70% (11722/16729) 2024-08-20T22:24:02.0986369Z Resolving deltas: 71% (11882/16729) 2024-08-20T22:24:02.0996526Z Resolving deltas: 72% (12048/16729) 2024-08-20T22:24:02.1008399Z Resolving deltas: 73% (12218/16729) 2024-08-20T22:24:02.1016334Z Resolving deltas: 74% (12387/16729) 2024-08-20T22:24:02.1030170Z Resolving deltas: 75% (12551/16729) 2024-08-20T22:24:02.1035360Z Resolving deltas: 76% (12778/16729) 2024-08-20T22:24:02.1044420Z Resolving deltas: 77% (12887/16729) 2024-08-20T22:24:02.1057429Z Resolving deltas: 78% (13049/16729) 2024-08-20T22:24:02.1064410Z Resolving deltas: 79% (13252/16729) 2024-08-20T22:24:02.1074084Z Resolving deltas: 80% (13384/16729) 2024-08-20T22:24:02.1083612Z Resolving deltas: 81% (13568/16729) 2024-08-20T22:24:02.1092769Z Resolving deltas: 82% (13739/16729) 2024-08-20T22:24:02.1101438Z Resolving deltas: 83% (13887/16729) 2024-08-20T22:24:02.1110633Z Resolving deltas: 84% (14080/16729) 2024-08-20T22:24:02.1116596Z Resolving deltas: 85% (14256/16729) 2024-08-20T22:24:02.1125643Z Resolving deltas: 86% (14400/16729) 2024-08-20T22:24:02.1132929Z Resolving deltas: 87% (14568/16729) 2024-08-20T22:24:02.1143230Z Resolving deltas: 88% (14725/16729) 2024-08-20T22:24:02.1149139Z Resolving deltas: 89% (14904/16729) 2024-08-20T22:24:02.1157981Z Resolving deltas: 90% (15068/16729) 2024-08-20T22:24:02.1167118Z Resolving deltas: 91% (15224/16729) 2024-08-20T22:24:02.1174094Z Resolving deltas: 92% (15393/16729) 2024-08-20T22:24:02.1182743Z Resolving deltas: 93% (15563/16729) 2024-08-20T22:24:02.1192738Z Resolving deltas: 94% (15726/16729) 2024-08-20T22:24:02.1200623Z Resolving deltas: 95% (15919/16729) 2024-08-20T22:24:02.1207473Z Resolving deltas: 96% (16081/16729) 2024-08-20T22:24:02.1217545Z Resolving deltas: 97% (16235/16729) 2024-08-20T22:24:02.1267780Z Resolving deltas: 98% (16396/16729) 2024-08-20T22:24:02.1343910Z Resolving deltas: 99% (16567/16729) 2024-08-20T22:24:02.1344445Z Resolving deltas: 100% (16729/16729) 2024-08-20T22:24:02.1344872Z Resolving deltas: 100% (16729/16729), done. 2024-08-20T22:24:03.1363835Z + pushd torchbench 2024-08-20T22:24:03.1368046Z ~/workspace/torchbench ~/workspace 2024-08-20T22:24:03.1368576Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:24:03.2166357Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T22:24:03.2166921Z 2024-08-20T22:24:03.2167475Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T22:24:03.2168289Z changes and commit them, and you can discard any commits you make in this 2024-08-20T22:24:03.2169038Z state without impacting any branches by switching back to a branch. 2024-08-20T22:24:03.2169476Z 2024-08-20T22:24:03.2169783Z If you want to create a new branch to retain commits you create, you may 2024-08-20T22:24:03.2170605Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T22:24:03.2171022Z 2024-08-20T22:24:03.2171236Z git switch -c 2024-08-20T22:24:03.2171504Z 2024-08-20T22:24:03.2171641Z Or undo this operation with: 2024-08-20T22:24:03.2171905Z 2024-08-20T22:24:03.2172022Z git switch - 2024-08-20T22:24:03.2172208Z 2024-08-20T22:24:03.2172595Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T22:24:03.2173072Z 2024-08-20T22:24:03.2173247Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T22:24:03.2178266Z + '[' '' ']' 2024-08-20T22:24:03.2178821Z + python install.py --continue_on_fail 2024-08-20T22:24:05.8886684Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T22:24:24.1864768Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T22:24:27.9210427Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T22:24:29.8232076Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T22:24:31.4481676Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T22:24:31.4716326Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T22:24:39.1163655Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T22:24:41.8438416Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T22:24:44.5570300Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T22:24:47.3250511Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T22:24:47.3253329Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T22:24:48.3666697Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T22:24:51.0619986Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T22:24:51.0856274Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T22:25:47.4221190Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T22:25:55.8480346Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T22:26:02.9263166Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T22:26:09.7307464Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T22:26:19.5606122Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T22:26:26.4313623Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T22:26:33.4008839Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T22:26:40.5275198Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T22:26:47.5384906Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T22:26:55.0114838Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T22:27:01.9035176Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T22:27:09.0529920Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T22:27:10.2638426Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T22:27:46.2586349Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T22:27:50.7813633Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T22:27:56.0025760Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:28:06.8888033Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:28:08.0747185Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:28:09.5573773Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:28:16.1327560Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:28:23.2106872Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:28:28.7522088Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:28:36.4231169Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:28:42.3157829Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:28:47.3288083Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:28:53.5162395Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:29:10.0220985Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:29:16.2391982Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:29:20.1383520Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:29:24.8918215Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:29:31.7318018Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:29:31.7320808Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:29:44.8010789Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:29:48.7265332Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:29:48.7267929Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:29:56.3132006Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:29:57.5328450Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:29:58.7657725Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:30:43.4779582Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:32:12.6731162Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:32:12.6975753Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:32:14.1993788Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:32:14.2235442Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:32:14.2477811Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:32:14.2726053Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:32:14.2963963Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:32:14.3206056Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:32:35.4622934Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:32:35.4625759Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:32:36.7169832Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:32:39.1112399Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:32:39.1356233Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:32:39.1599290Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:32:39.1845505Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:32:39.2092863Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:32:39.2340324Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:32:45.8579620Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:32:47.3447766Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:32:50.8192836Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:32:50.8439751Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:32:50.8681349Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:32:50.8923477Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:32:50.9162115Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:32:50.9406218Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:33:08.6700767Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:33:24.6465087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:33:24.6707954Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:33:24.6711669Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:33:24.6715344Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:33:27.4308306Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:33:29.1997070Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:33:29.2249406Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:33:48.0984334Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:33:54.6852167Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:33:59.4038336Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:34:06.4811332Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:34:06.5057563Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:34:06.5302621Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:34:06.5550033Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:34:06.5792745Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:34:06.6041183Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:34:06.6283572Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:34:06.6526183Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:34:14.4949416Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:34:26.8278509Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:34:26.8529692Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:34:28.9933597Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:34:30.9816482Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:34:31.4751900Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:34:31.4752687Z Print all dependencies after TorchBench is installed 2024-08-20T22:34:31.4755067Z + python -mpip freeze 2024-08-20T22:34:32.0194499Z absl-py==2.1.0 2024-08-20T22:34:32.0194993Z accelerate==0.33.0 2024-08-20T22:34:32.0195354Z aiohappyeyeballs==2.4.0 2024-08-20T22:34:32.0195703Z aiohttp==3.10.5 2024-08-20T22:34:32.0196108Z aiosignal==1.3.1 2024-08-20T22:34:32.0196576Z annotated-types==0.7.0 2024-08-20T22:34:32.0196988Z antlr4-python3-runtime==4.9.3 2024-08-20T22:34:32.0197352Z anyascii==0.3.2 2024-08-20T22:34:32.0197647Z astroid==3.2.4 2024-08-20T22:34:32.0197961Z astunparse==1.6.3 2024-08-20T22:34:32.0198721Z async-timeout==4.0.3 2024-08-20T22:34:32.0199613Z attrs==23.1.0 2024-08-20T22:34:32.0200007Z audioread==3.0.1 2024-08-20T22:34:32.0200376Z babel==2.16.0 2024-08-20T22:34:32.0200781Z beautifulsoup4==4.12.3 2024-08-20T22:34:32.0207548Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:34:32.0208712Z bibtexparser==2.0.0b7 2024-08-20T22:34:32.0209116Z black==24.8.0 2024-08-20T22:34:32.0209510Z blinker==1.8.2 2024-08-20T22:34:32.0209897Z blis==0.7.11 2024-08-20T22:34:32.0210211Z blobfile==2.1.1 2024-08-20T22:34:32.0210505Z bokeh==3.5.1 2024-08-20T22:34:32.0210843Z boto3==1.19.12 2024-08-20T22:34:32.0211200Z botocore==1.22.12 2024-08-20T22:34:32.0211984Z bs4==0.0.2 2024-08-20T22:34:32.0212318Z cachetools==5.5.0 2024-08-20T22:34:32.0212658Z cardboardlint==1.3.1 2024-08-20T22:34:32.0213062Z catalogue==2.0.10 2024-08-20T22:34:32.0213389Z certifi==2024.7.4 2024-08-20T22:34:32.0213778Z cffi==1.17.0 2024-08-20T22:34:32.0214235Z charset-normalizer==3.3.2 2024-08-20T22:34:32.0214660Z click==8.1.7 2024-08-20T22:34:32.0214978Z clldutils==3.22.2 2024-08-20T22:34:32.0215278Z cloudpathlib==0.18.1 2024-08-20T22:34:32.0215671Z cloudpickle==3.0.0 2024-08-20T22:34:32.0215981Z colorama==0.4.6 2024-08-20T22:34:32.0216312Z colorlog==6.8.2 2024-08-20T22:34:32.0216639Z confection==0.1.5 2024-08-20T22:34:32.0216942Z contourpy==1.2.1 2024-08-20T22:34:32.0217314Z coremltools==5.0b5 2024-08-20T22:34:32.0217623Z cryptography==43.0.0 2024-08-20T22:34:32.0217957Z csvw==3.3.0 2024-08-20T22:34:32.0218285Z cycler==0.12.1 2024-08-20T22:34:32.0218572Z cymem==2.0.8 2024-08-20T22:34:32.0218969Z DALL-E==0.1 2024-08-20T22:34:32.0219262Z datasets==2.21.0 2024-08-20T22:34:32.0219575Z decorator==5.1.1 2024-08-20T22:34:32.0219965Z defusedxml==0.7.1 2024-08-20T22:34:32.0220319Z Deprecated==1.2.14 2024-08-20T22:34:32.0221213Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:34:32.0222161Z diffusers==0.20.2 2024-08-20T22:34:32.0222563Z dill==0.3.8 2024-08-20T22:34:32.0222941Z diskcache==5.6.3 2024-08-20T22:34:32.0223282Z distro==1.9.0 2024-08-20T22:34:32.0223560Z dlinfo==1.2.1 2024-08-20T22:34:32.0223945Z docker-pycreds==0.4.0 2024-08-20T22:34:32.0224356Z dominate==2.9.1 2024-08-20T22:34:32.0225173Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:34:32.0226018Z einops==0.8.0 2024-08-20T22:34:32.0226344Z exceptiongroup==1.2.2 2024-08-20T22:34:32.0226766Z execnet==2.1.1 2024-08-20T22:34:32.0227147Z expecttest==0.1.6 2024-08-20T22:34:32.0227658Z FastNLP==0.6.0 2024-08-20T22:34:32.0228119Z ffmpeg-python==0.2.0 2024-08-20T22:34:32.0228457Z filelock==3.13.1 2024-08-20T22:34:32.0228751Z Flask==3.0.3 2024-08-20T22:34:32.0229035Z flatbuffers==2.0 2024-08-20T22:34:32.0229327Z fonttools==4.53.1 2024-08-20T22:34:32.0229624Z frozenlist==1.4.1 2024-08-20T22:34:32.0229921Z fsspec==2024.6.1 2024-08-20T22:34:32.0230201Z ftfy==6.2.3 2024-08-20T22:34:32.0230820Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:34:32.0231518Z future==1.0.0 2024-08-20T22:34:32.0231812Z fvcore==0.1.5.post20221221 2024-08-20T22:34:32.0232143Z gdown==5.2.0 2024-08-20T22:34:32.0232429Z geojson==2.5.0 2024-08-20T22:34:32.0232715Z ghstack==0.8.0 2024-08-20T22:34:32.0233004Z gitdb==4.0.11 2024-08-20T22:34:32.0233282Z GitPython==3.1.43 2024-08-20T22:34:32.0233634Z google-auth==2.34.0 2024-08-20T22:34:32.0234005Z google-auth-oauthlib==1.0.0 2024-08-20T22:34:32.0234358Z grpcio==1.65.5 2024-08-20T22:34:32.0234643Z gym==0.26.2 2024-08-20T22:34:32.0234984Z gym-notices==0.0.8 2024-08-20T22:34:32.0235315Z h5py==3.11.0 2024-08-20T22:34:32.0235592Z higher==0.2.1 2024-08-20T22:34:32.0235934Z huggingface-hub==0.24.6 2024-08-20T22:34:32.0236299Z hydra-core==1.3.2 2024-08-20T22:34:32.0236591Z hypothesis==5.35.1 2024-08-20T22:34:32.0236888Z idna==3.7 2024-08-20T22:34:32.0237534Z imageio==2.35.1 2024-08-20T22:34:32.0238154Z importlib_metadata==8.4.0 2024-08-20T22:34:32.0238496Z inflect==7.3.1 2024-08-20T22:34:32.0238789Z iniconfig==2.0.0 2024-08-20T22:34:32.0239083Z iopath==0.1.9 2024-08-20T22:34:32.0239390Z isodate==0.6.1 2024-08-20T22:34:32.0239714Z isort==5.13.2 2024-08-20T22:34:32.0240005Z itsdangerous==2.2.0 2024-08-20T22:34:32.0240310Z Jinja2==3.1.4 2024-08-20T22:34:32.0240586Z jmespath==0.10.0 2024-08-20T22:34:32.0240877Z joblib==1.4.2 2024-08-20T22:34:32.0241157Z jsonpatch==1.33 2024-08-20T22:34:32.0241444Z jsonpointer==3.0.0 2024-08-20T22:34:32.0241745Z jsonschema==4.23.0 2024-08-20T22:34:32.0242160Z jsonschema-specifications==2023.12.1 2024-08-20T22:34:32.0242568Z junitparser==2.1.1 2024-08-20T22:34:32.0242904Z kaldi-io==0.9.8 2024-08-20T22:34:32.0243355Z kiwisolver==1.4.5 2024-08-20T22:34:32.0243662Z kornia==0.7.3 2024-08-20T22:34:32.0243944Z kornia_rs==0.1.5 2024-08-20T22:34:32.0244233Z lameenc==1.7.0 2024-08-20T22:34:32.0244518Z langcodes==3.4.0 2024-08-20T22:34:32.0244812Z langdetect==1.0.9 2024-08-20T22:34:32.0245207Z language-tags==1.2.0 2024-08-20T22:34:32.0245530Z language_data==1.2.0 2024-08-20T22:34:32.0245838Z lark==0.12.0 2024-08-20T22:34:32.0246115Z lazy_loader==0.4 2024-08-20T22:34:32.0246400Z libcst==1.4.0 2024-08-20T22:34:32.0246680Z librosa==0.9.2 2024-08-20T22:34:32.0246965Z lintrunner==0.12.5 2024-08-20T22:34:32.0247257Z llvmlite==0.38.1 2024-08-20T22:34:32.0247545Z lxml==4.9.4 2024-08-20T22:34:32.0247867Z marisa-trie==1.2.0 2024-08-20T22:34:32.0248166Z Markdown==3.7 2024-08-20T22:34:32.0248493Z markdown-it-py==3.0.0 2024-08-20T22:34:32.0248823Z MarkupSafe==2.1.5 2024-08-20T22:34:32.0249128Z matplotlib==3.8.4 2024-08-20T22:34:32.0249418Z mccabe==0.7.0 2024-08-20T22:34:32.0249696Z mdurl==0.1.2 2024-08-20T22:34:32.0250024Z ml-dtypes==0.4.0 2024-08-20T22:34:32.0250320Z MonkeyType==23.3.0 2024-08-20T22:34:32.0250664Z more-itertools==10.4.0 2024-08-20T22:34:32.0250983Z mpmath==1.3.0 2024-08-20T22:34:32.0251274Z msgpack==1.0.8 2024-08-20T22:34:32.0251606Z multidict==6.0.5 2024-08-20T22:34:32.0251933Z multiprocess==0.70.16 2024-08-20T22:34:32.0252255Z murmurhash==1.0.10 2024-08-20T22:34:32.0252551Z musdb==0.4.2 2024-08-20T22:34:32.0252867Z museval==0.4.1 2024-08-20T22:34:32.0253156Z mypy==1.10.0 2024-08-20T22:34:32.0253499Z mypy-extensions==1.0.0 2024-08-20T22:34:32.0253816Z networkx==2.8.8 2024-08-20T22:34:32.0254103Z ninja==1.10.2 2024-08-20T22:34:32.0254386Z nose==1.3.7 2024-08-20T22:34:32.0254658Z numba==0.55.2 2024-08-20T22:34:32.0254928Z numpy==1.21.2 2024-08-20T22:34:32.0255284Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:34:32.0255708Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:34:32.0256144Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:34:32.0256598Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:34:32.0257052Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:34:32.0257453Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:34:32.0257855Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:34:32.0258276Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:34:32.0258718Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:34:32.0259143Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:34:32.0259536Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:34:32.0259955Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:34:32.0260305Z oauthlib==3.2.2 2024-08-20T22:34:32.0260595Z omegaconf==2.3.0 2024-08-20T22:34:32.0260887Z onnx==1.16.1 2024-08-20T22:34:32.0261180Z onnxscript==0.1.0.dev20240817 2024-08-20T22:34:32.0261530Z opacus==1.5.2 2024-08-20T22:34:32.0261864Z opencv-python==4.8.0.74 2024-08-20T22:34:32.0262227Z opt-einsum==3.3.0 2024-08-20T22:34:32.0262532Z optree==0.12.1 2024-08-20T22:34:32.0262817Z packaging==24.1 2024-08-20T22:34:32.0263114Z pandas==2.0.3 2024-08-20T22:34:32.0263395Z patch==1.16 2024-08-20T22:34:32.0263669Z pathspec==0.12.1 2024-08-20T22:34:32.0263965Z phonemizer==3.3.0 2024-08-20T22:34:32.0264262Z pillow==10.3.0 2024-08-20T22:34:32.0264553Z platformdirs==4.2.2 2024-08-20T22:34:32.0264852Z pluggy==1.5.0 2024-08-20T22:34:32.0265130Z pooch==1.8.2 2024-08-20T22:34:32.0265415Z portalocker==2.10.1 2024-08-20T22:34:32.0265718Z preshed==3.0.9 2024-08-20T22:34:32.0266183Z prettytable==3.11.0 2024-08-20T22:34:32.0266491Z protobuf==3.20.2 2024-08-20T22:34:32.0266785Z psutil==6.0.0 2024-08-20T22:34:32.0267113Z py-cpuinfo==9.0.0 2024-08-20T22:34:32.0267549Z pyaml==24.7.0 2024-08-20T22:34:32.0267835Z pyarrow==17.0.0 2024-08-20T22:34:32.0268123Z pyasn1==0.6.0 2024-08-20T22:34:32.0268406Z pyasn1_modules==0.4.0 2024-08-20T22:34:32.0268731Z pyclipper==1.3.0.post5 2024-08-20T22:34:32.0269057Z pycocotools==2.0.8 2024-08-20T22:34:32.0269362Z pycparser==2.22 2024-08-20T22:34:32.0269658Z pycryptodomex==3.20.0 2024-08-20T22:34:32.0269983Z pydantic==2.8.2 2024-08-20T22:34:32.0270278Z pydantic_core==2.20.1 2024-08-20T22:34:32.0270585Z pydot==3.0.1 2024-08-20T22:34:32.0270971Z pygame==2.6.0 2024-08-20T22:34:32.0271260Z PyGithub==2.3.0 2024-08-20T22:34:32.0271553Z Pygments==2.15.0 2024-08-20T22:34:32.0271840Z PyJWT==2.9.0 2024-08-20T22:34:32.0272123Z pylatexenc==2.10 2024-08-20T22:34:32.0272416Z pylint==3.2.6 2024-08-20T22:34:32.0272689Z PyNaCl==1.5.0 2024-08-20T22:34:32.0272976Z pynvml==11.5.3 2024-08-20T22:34:32.0273271Z pyparsing==3.1.2 2024-08-20T22:34:32.0273571Z pypdfium2==4.30.0 2024-08-20T22:34:32.0273860Z pysbd==0.3.4 2024-08-20T22:34:32.0274141Z PySocks==1.7.1 2024-08-20T22:34:32.0274424Z pytest==7.3.2 2024-08-20T22:34:32.0274763Z pytest-benchmark==4.0.0 2024-08-20T22:34:32.0275181Z pytest-cpp==2.3.0 2024-08-20T22:34:32.0275535Z pytest-flakefinder==1.1.0 2024-08-20T22:34:32.0275939Z pytest-rerunfailures==14.0 2024-08-20T22:34:32.0276318Z pytest-xdist==3.3.1 2024-08-20T22:34:32.0276683Z python-dateutil==2.9.0.post0 2024-08-20T22:34:32.0277472Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:34:32.0278890Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:34:32.0279838Z pytz==2024.1 2024-08-20T22:34:32.0280198Z PyWavelets==1.4.1 2024-08-20T22:34:32.0280575Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:34:32.0281023Z rapidfuzz==3.9.6 2024-08-20T22:34:32.0281317Z rdflib==7.0.0 2024-08-20T22:34:32.0281589Z redis==5.0.8 2024-08-20T22:34:32.0281871Z referencing==0.35.1 2024-08-20T22:34:32.0282174Z regex==2024.7.24 2024-08-20T22:34:32.0282464Z requests==2.32.3 2024-08-20T22:34:32.0282814Z requests-oauthlib==2.0.0 2024-08-20T22:34:32.0283155Z resampy==0.4.3 2024-08-20T22:34:32.0283437Z rfc3986==1.5.0 2024-08-20T22:34:32.0283710Z rich==13.7.1 2024-08-20T22:34:32.0283985Z rockset==1.0.3 2024-08-20T22:34:32.0284307Z rpds-py==0.20.0 2024-08-20T22:34:32.0284585Z rsa==4.9 2024-08-20T22:34:32.0284853Z s3transfer==0.5.2 2024-08-20T22:34:32.0285151Z safetensors==0.4.4 2024-08-20T22:34:32.0285494Z scikit-image==0.21.0 2024-08-20T22:34:32.0285844Z scikit-learn==1.5.1 2024-08-20T22:34:32.0286149Z scipy==1.10.1 2024-08-20T22:34:32.0286429Z segments==2.2.1 2024-08-20T22:34:32.0286715Z sentencepiece==0.2.0 2024-08-20T22:34:32.0287067Z sentry-sdk==2.13.0 2024-08-20T22:34:32.0287371Z setproctitle==1.3.3 2024-08-20T22:34:32.0287682Z shapely==2.0.6 2024-08-20T22:34:32.0287966Z shellingham==1.5.4 2024-08-20T22:34:32.0288271Z simplejson==3.19.3 2024-08-20T22:34:32.0288661Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:34:32.0289155Z smart-open==7.0.4 2024-08-20T22:34:32.0289453Z smmap==5.0.1 2024-08-20T22:34:32.0289747Z sortedcontainers==2.4.0 2024-08-20T22:34:32.0290074Z soundfile==0.12.1 2024-08-20T22:34:32.0290366Z soupsieve==2.6 2024-08-20T22:34:32.0290650Z soxr==0.4.0 2024-08-20T22:34:32.0290926Z spacy==3.7.6 2024-08-20T22:34:32.0291239Z spacy-legacy==3.0.12 2024-08-20T22:34:32.0291594Z spacy-loggers==1.0.5 2024-08-20T22:34:32.0291906Z srsly==2.4.8 2024-08-20T22:34:32.0292186Z stempeg==0.2.3 2024-08-20T22:34:32.0292470Z submitit==1.5.1 2024-08-20T22:34:32.0292758Z sympy==1.13.1 2024-08-20T22:34:32.0293044Z tabulate==0.9.0 2024-08-20T22:34:32.0293380Z tb-nightly==2.13.0a20230426 2024-08-20T22:34:32.0293723Z tensorboard==2.13.0 2024-08-20T22:34:32.0294097Z tensorboard-data-server==0.7.2 2024-08-20T22:34:32.0294623Z tensorboardX==2.6.2.2 2024-08-20T22:34:32.0294940Z termcolor==2.4.0 2024-08-20T22:34:32.0295234Z thinc==8.2.5 2024-08-20T22:34:32.0295522Z threadpoolctl==3.5.0 2024-08-20T22:34:32.0295835Z tifffile==2024.8.10 2024-08-20T22:34:32.0296627Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:34:32.0297388Z tlparse==0.3.25 2024-08-20T22:34:32.0297682Z tokenizers==0.15.2 2024-08-20T22:34:32.0297974Z tomli==2.0.1 2024-08-20T22:34:32.0298254Z tomlkit==0.13.2 2024-08-20T22:34:32.0299602Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b70a70c0c1fc7cc3f43c6342d758f4ffd3a540a88b786d266604d87618be5127 2024-08-20T22:34:32.0301213Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:34:32.0302258Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:34:32.0303224Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:34:32.0304211Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:34:32.0305359Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:34:32.0306439Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:34:32.0307538Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:34:32.0308231Z tornado==6.4.1 2024-08-20T22:34:32.0308519Z tqdm==4.66.5 2024-08-20T22:34:32.0308812Z transformers==4.38.1 2024-08-20T22:34:32.0309122Z treetable==0.2.5 2024-08-20T22:34:32.0309928Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:34:32.0310701Z typeguard==4.3.0 2024-08-20T22:34:32.0311001Z typer==0.12.4 2024-08-20T22:34:32.0311295Z typing_extensions==4.12.2 2024-08-20T22:34:32.0311631Z tzdata==2024.1 2024-08-20T22:34:32.0311920Z Unidecode==1.3.8 2024-08-20T22:34:32.0312285Z unittest-xml-reporting==3.2.0 2024-08-20T22:34:32.0312655Z uritemplate==4.1.1 2024-08-20T22:34:32.0312959Z urllib3==1.26.19 2024-08-20T22:34:32.0313250Z visdom==0.2.4 2024-08-20T22:34:32.0313522Z wandb==0.17.7 2024-08-20T22:34:32.0313813Z wasabi==1.1.3 2024-08-20T22:34:32.0314094Z wcwidth==0.2.13 2024-08-20T22:34:32.0314377Z weasel==0.4.1 2024-08-20T22:34:32.0314708Z websocket-client==1.8.0 2024-08-20T22:34:32.0315043Z Werkzeug==3.0.3 2024-08-20T22:34:32.0315333Z wrapt==1.16.0 2024-08-20T22:34:32.0315607Z xdoctest==1.1.0 2024-08-20T22:34:32.0315895Z xxhash==3.5.0 2024-08-20T22:34:32.0316182Z xyzservices==2024.6.0 2024-08-20T22:34:32.0316491Z yacs==0.1.8 2024-08-20T22:34:32.0316761Z yarl==1.9.4 2024-08-20T22:34:32.0317081Z z3-solver==4.12.2.0 2024-08-20T22:34:32.0317381Z zipp==3.20.0 2024-08-20T22:34:32.0518025Z + popd 2024-08-20T22:34:32.0518419Z ~/workspace 2024-08-20T22:34:32.0518827Z + [[ dynamic_aot_eager_torchbench != *cpu* ]] 2024-08-20T22:34:32.0519311Z + install_torchrec_and_fbgemm 2024-08-20T22:34:32.0519739Z + local torchrec_commit 2024-08-20T22:34:32.0522523Z ++ get_pinned_commit torchrec 2024-08-20T22:34:32.0522955Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-08-20T22:34:32.0542541Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T22:34:32.0543217Z + local fbgemm_commit 2024-08-20T22:34:32.0547344Z ++ get_pinned_commit fbgemm 2024-08-20T22:34:32.0548011Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-08-20T22:34:32.0565872Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:32.0566718Z + pip_uninstall torchrec-nightly 2024-08-20T22:34:32.0567185Z + pip uninstall -y torchrec-nightly 2024-08-20T22:34:32.4358610Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-08-20T22:34:32.4719468Z + pip_uninstall fbgemm-gpu-nightly 2024-08-20T22:34:32.4720426Z + pip uninstall -y fbgemm-gpu-nightly 2024-08-20T22:34:32.8535415Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-08-20T22:34:32.8903309Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:34:32.8904269Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:34:33.3607763Z Collecting setuptools-git-versioning 2024-08-20T22:34:33.3804140Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-08-20T22:34:33.5004547Z Collecting scikit-build 2024-08-20T22:34:33.5043356Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-08-20T22:34:33.6252245Z Collecting pyre-extensions 2024-08-20T22:34:33.6289632Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-08-20T22:34:33.6807990Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-08-20T22:34:33.6811381Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0) 2024-08-20T22:34:33.7384881Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-08-20T22:34:33.7440024Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-08-20T22:34:33.7917055Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-08-20T22:34:33.7922118Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-08-20T22:34:33.7926082Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-08-20T22:34:33.8469694Z Collecting typing-inspect (from pyre-extensions) 2024-08-20T22:34:33.8506164Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-08-20T22:34:33.8980731Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-08-20T22:34:33.9060104Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-08-20T22:34:33.9105591Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-08-20T22:34:33.9816135Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-08-20T22:34:34.0346829Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-08-20T22:34:34.0925810Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-08-20T22:34:34.1485800Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-08-20T22:34:35.0475221Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-08-20T22:34:35.4019181Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-08-20T22:34:35.5118030Z + CUDA_PATH=/usr/local/cuda-12.1 2024-08-20T22:34:35.5119479Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:34:35.5121333Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:34:35.8815517Z Collecting fbgemm-gpu 2024-08-20T22:34:35.8818604Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-ndli_p_b/fbgemm-gpu_5f509485b9e84081a219713b711573d5 2024-08-20T22:34:35.8849090Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-ndli_p_b/fbgemm-gpu_5f509485b9e84081a219713b711573d5 2024-08-20T22:34:37.0218737Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-08-20T22:34:37.0247552Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:37.5240937Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:37.9930749Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:37.9931872Z Running command git submodule update --init --recursive -q 2024-08-20T22:34:47.6710889Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:34:47.6730804Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-08-20T22:34:47.6738940Z Building wheels for collected packages: fbgemm-gpu 2024-08-20T23:19:19.4189866Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T23:19:19.6027672Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252620091 sha256=f2e1cd711f8477df098798f9a7f655febe57ba32717f42727b48704aebfcd2a7 2024-08-20T23:19:19.6029353Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-08-20T23:19:19.6052362Z Successfully built fbgemm-gpu 2024-08-20T23:19:20.4493829Z Installing collected packages: fbgemm-gpu 2024-08-20T23:19:23.8509461Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-08-20T23:19:24.7694013Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:24.7695566Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:25.1413295Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:25.1417028Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-sjtr3mve 2024-08-20T23:19:25.1446373Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-sjtr3mve 2024-08-20T23:19:25.9078608Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-08-20T23:19:25.9107507Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:26.2166158Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:26.5785304Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:26.8578083Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T23:19:26.8614477Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-08-20T23:19:26.8618075Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-08-20T23:19:26.8620375Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-08-20T23:19:26.9105064Z Collecting torchmetrics (from torchrec==0.3.2) 2024-08-20T23:19:26.9314578Z Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB) 2024-08-20T23:19:26.9705575Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5) 2024-08-20T23:19:26.9707589Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-08-20T23:19:26.9718277Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-08-20T23:19:26.9727363Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1) 2024-08-20T23:19:26.9989519Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-08-20T23:19:26.9992765Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:19:26.9996155Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:19:27.0018115Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-08-20T23:19:27.0022396Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6) 2024-08-20T23:19:27.0166170Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-08-20T23:19:27.0217105Z Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) 2024-08-20T23:19:27.0306013Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0) 2024-08-20T23:19:27.0309636Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-08-20T23:19:27.0329965Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-08-20T23:19:27.0363200Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-08-20T23:19:27.0366686Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-08-20T23:19:27.0370248Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-08-20T23:19:27.0372591Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1) 2024-08-20T23:19:27.0377614Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1) 2024-08-20T23:19:27.0392897Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-08-20T23:19:27.0799838Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-08-20T23:19:27.0947918Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB) 2024-08-20T23:19:27.1187679Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB) 2024-08-20T23:19:27.1285875Z Building wheels for collected packages: torchrec 2024-08-20T23:19:27.6506082Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-08-20T23:19:27.6514645Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=d2a51905d42e3d4db7f25483303a0f8399ab95f1dc646418fc1ed32e8bbbc398 2024-08-20T23:19:27.6516495Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-08-20T23:19:27.6542199Z Successfully built torchrec 2024-08-20T23:19:28.4360621Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-08-20T23:19:29.0499968Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2 2024-08-20T23:19:29.1921035Z ++ pwd 2024-08-20T23:19:29.1923214Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T23:19:29.1923734Z + test_dynamo_benchmark torchbench 1 2024-08-20T23:19:29.1928470Z ++ pwd 2024-08-20T23:19:29.1932238Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:19:29.1932789Z + local suite=torchbench 2024-08-20T23:19:29.1933130Z + shift 2024-08-20T23:19:29.1936368Z + local shard_id=1 2024-08-20T23:19:29.1936748Z + shift 2024-08-20T23:19:29.1937475Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]] 2024-08-20T23:19:29.1938283Z + [[ dynamic_aot_eager_torchbench == *perf* ]] 2024-08-20T23:19:29.1938812Z + [[ dynamic_aot_eager_torchbench == *cpu* ]] 2024-08-20T23:19:29.1939344Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]] 2024-08-20T23:19:29.1940840Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-08-20T23:19:29.1941502Z ++ pwd 2024-08-20T23:19:29.1942230Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:19:29.1942910Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:19:29.1983964Z + local name=inference 2024-08-20T23:19:29.1984404Z + shift 2024-08-20T23:19:29.1984776Z + local suite=torchbench 2024-08-20T23:19:29.1985114Z + shift 2024-08-20T23:19:29.1985406Z + local shard_id=1 2024-08-20T23:19:29.1985802Z + shift 2024-08-20T23:19:29.1986104Z + partition_flags=() 2024-08-20T23:19:29.1986444Z + local partition_flags 2024-08-20T23:19:29.1986901Z + [[ -n 2 ]] 2024-08-20T23:19:29.1987243Z + [[ -n 1 ]] 2024-08-20T23:19:29.1987943Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:19:29.1988735Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]] 2024-08-20T23:19:29.1989338Z + [[ dynamic_aot_eager_torchbench == *perf* ]] 2024-08-20T23:19:29.1989867Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]] 2024-08-20T23:19:29.1990438Z + [[ dynamic_aot_eager_torchbench == *_avx2* ]] 2024-08-20T23:19:29.1990941Z + [[ dynamic_aot_eager_torchbench == *_avx512* ]] 2024-08-20T23:19:29.1992806Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T23:19:35.4179171Z 2024-08-20T23:19:36.2161303Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:19:36.2161954Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:19:36.2194805Z cuda eval llama 2024-08-20T23:20:08.6066251Z pass 2024-08-20T23:20:08.6483139Z TIMING: backend_compile:23.72078 entire_frame_compile:31.75419 total_wall_time:31.75419 2024-08-20T23:20:08.6484512Z STATS: call_* op count: 1058 | FakeTensorMode.__torch_dispatch__:26500 | attempt fast:840 | fast is_contiguous:840 | FakeTensor.__torch_dispatch__:64 | ProxyTorchDispatchMode.__torch_dispatch__:8227 2024-08-20T23:20:08.6485808Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique) 2024-08-20T23:20:13.7565034Z 2024-08-20T23:20:14.0591893Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:14.0592380Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:14.0592814Z cuda eval llama_v2_7b_16h 2024-08-20T23:20:14.0617843Z Traceback (most recent call last): 2024-08-20T23:20:14.0618555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:20:14.0619221Z ) = runner.load_model( 2024-08-20T23:20:14.0619993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:20:14.0620872Z benchmark = benchmark_cls( 2024-08-20T23:20:14.0621745Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:20:14.0622805Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:20:14.0623793Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-08-20T23:20:14.0625009Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:14.0626375Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:20:14.0627683Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:14.0628656Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:20:14.0629397Z self._skip_by_device_name() 2024-08-20T23:20:14.0630197Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:20:14.0631420Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:20:14.0632550Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-08-20T23:20:14.0633159Z 2024-08-20T23:20:14.0633284Z model_fail_to_load 2024-08-20T23:20:17.5158177Z 2024-08-20T23:20:17.8198890Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:17.8199394Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:17.8199841Z cuda eval llava 2024-08-20T23:20:17.8206737Z Traceback (most recent call last): 2024-08-20T23:20:17.8207639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:20:17.8208524Z ) = runner.load_model( 2024-08-20T23:20:17.8209297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:20:17.8210013Z benchmark = benchmark_cls( 2024-08-20T23:20:17.8210733Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:20:17.8211514Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:20:17.8212357Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:20:17.8213455Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:17.8214742Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:20:17.8215912Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:17.8216884Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:20:17.8217868Z self._skip_by_device_name() 2024-08-20T23:20:17.8218663Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:20:17.8219880Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:20:17.8220960Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:20:17.8221521Z 2024-08-20T23:20:17.8221649Z model_fail_to_load 2024-08-20T23:20:21.2805068Z 2024-08-20T23:20:22.0964887Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:22.0965372Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:22.0972538Z cuda eval maml 2024-08-20T23:20:22.0974706Z pass_due_to_skip 2024-08-20T23:20:22.0983055Z TIMING: 2024-08-20T23:20:22.0983418Z STATS: call_* op count: 0 2024-08-20T23:20:22.0983965Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:20:25.6732825Z 2024-08-20T23:20:26.1957641Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:26.1958226Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:26.1966058Z cuda eval maml_omniglot 2024-08-20T23:20:27.0959749Z pass 2024-08-20T23:20:27.0960530Z TIMING: backend_compile:0.31744 entire_frame_compile:0.69845 total_wall_time:0.69845 2024-08-20T23:20:27.0961645Z STATS: call_* op count: 28 | FakeTensorMode.__torch_dispatch__:1267 | ProxyTorchDispatchMode.__torch_dispatch__:328 2024-08-20T23:20:27.0962589Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-08-20T23:20:30.7338975Z 2024-08-20T23:20:30.9494588Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-08-20T23:20:30.9691518Z 2024-08-20T23:20:30.9691527Z 2024-08-20T23:20:31.0296915Z 0% 0.00/16.9M [00:00 will be ignored 2024-08-20T23:21:34.2306600Z [rank0]:W0820 23:21:34.229000 32794 torch/_dynamo/backends/distributed.py:89] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:21:34.2559544Z [rank0]:W0820 23:21:34.255000 32794 torch/_dynamo/backends/distributed.py:106] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:21:34.2615982Z [rank0]:W0820 23:21:34.255000 32794 torch/_dynamo/backends/distributed.py:106] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:21:34.2617480Z [rank0]:W0820 23:21:34.255000 32794 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:21:34.2619016Z [rank0]:W0820 23:21:34.255000 32794 torch/_dynamo/backends/distributed.py:106] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:21:34.2620427Z [rank0]:W0820 23:21:34.255000 32794 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ 0 │ 470 │ 44910720 │ 2024-08-20T23:21:34.2621831Z [rank0]:W0820 23:21:34.255000 32794 torch/_dynamo/backends/distributed.py:106] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:21:45.9121778Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:21:45.9123230Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:21:45.9124498Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] or: 2024-08-20T23:21:45.9125845Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:21:45.9127150Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] to include these operations in the captured graph. 2024-08-20T23:21:45.9128176Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:21:45.9129379Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break: from user code at: 2024-08-20T23:21:45.9131563Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:21:45.9133280Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] self._dequeue_and_enqueue(k) 2024-08-20T23:21:45.9135151Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:21:45.9136830Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] return func(*args, **kwargs) 2024-08-20T23:21:45.9138881Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:21:45.9140533Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] ptr = int(self.queue_ptr) 2024-08-20T23:21:45.9141495Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:21:45.9142273Z [rank0]:W0820 23:21:45.911000 32794 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:21:53.2048364Z pass 2024-08-20T23:21:53.2076182Z TIMING: entire_frame_compile:29.9855 backend_compile:9.9031 total_wall_time:29.9855 2024-08-20T23:21:53.2077601Z STATS: call_* op count: 879 | FakeTensorMode.__torch_dispatch__:32694 | FakeTensor.__torch_dispatch__:69 | ProxyTorchDispatchMode.__torch_dispatch__:8866 2024-08-20T23:21:53.2078733Z Dynamo produced 7 graphs covering 879 ops with 5 graph breaks (3 unique) 2024-08-20T23:21:53.8324977Z [rank0]:[W820 23:21:53.778065499 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:21:57.8960196Z 2024-08-20T23:21:58.2080728Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:21:58.2081320Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:21:58.2081863Z cuda eval moondream 2024-08-20T23:21:58.2088021Z Traceback (most recent call last): 2024-08-20T23:21:58.2088928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:21:58.2089786Z ) = runner.load_model( 2024-08-20T23:21:58.2090641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:21:58.2091594Z benchmark = benchmark_cls( 2024-08-20T23:21:58.2092577Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:21:58.2093400Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:21:58.2094291Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-08-20T23:21:58.2095720Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:21:58.2097358Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:21:58.2098544Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:21:58.2099517Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:21:58.2100255Z self._skip_by_device_name() 2024-08-20T23:21:58.2101053Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:21:58.2102503Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:21:58.2103596Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-08-20T23:21:58.2104168Z 2024-08-20T23:21:58.2104290Z model_fail_to_load 2024-08-20T23:22:01.6674179Z 2024-08-20T23:22:03.6503379Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:22:03.8967234Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:22:03.8968290Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:22:03.8969477Z using fused AdamW: True 2024-08-20T23:22:04.2936517Z 2024-08-20T23:22:04.2936905Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:22:04.2978660Z cuda eval nanogpt 2024-08-20T23:22:29.6434041Z pass 2024-08-20T23:22:29.6439582Z TIMING: backend_compile:16.97488 entire_frame_compile:24.74924 total_wall_time:24.74924 2024-08-20T23:22:29.6441203Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:28940 | attempt fast:1908 | fast is_contiguous:1908 | FakeTensor.__torch_dispatch__:216 | ProxyTorchDispatchMode.__torch_dispatch__:6106 2024-08-20T23:22:29.6442503Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:34.3475718Z 2024-08-20T23:22:36.7670651Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:36.7671135Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:22:36.7674408Z cuda eval nvidia_deeprecommender 2024-08-20T23:22:37.5991665Z pass 2024-08-20T23:22:37.6026403Z TIMING: backend_compile:0.19737 entire_frame_compile:0.52875 total_wall_time:0.52875 2024-08-20T23:22:37.6027766Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:731 | ProxyTorchDispatchMode.__torch_dispatch__:242 2024-08-20T23:22:37.6028720Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:41.2592937Z 2024-08-20T23:22:42.2062726Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:42.2063674Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:42.2080481Z cuda eval opacus_cifar10 2024-08-20T23:22:45.7097697Z pass 2024-08-20T23:22:45.7098539Z TIMING: backend_compile:0.8898 entire_frame_compile:3.14781 total_wall_time:3.14781 2024-08-20T23:22:45.7100065Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:2797 | ProxyTorchDispatchMode.__torch_dispatch__:1188 2024-08-20T23:22:45.7101436Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:49.4116345Z 2024-08-20T23:22:50.0631265Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:50.0631735Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:50.0694433Z cuda eval phlippe_densenet 2024-08-20T23:23:00.0351279Z pass 2024-08-20T23:23:00.0352658Z TIMING: backend_compile:4.26967 entire_frame_compile:9.58396 total_wall_time:9.58396 2024-08-20T23:23:00.0353806Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:18197 | ProxyTorchDispatchMode.__torch_dispatch__:4433 2024-08-20T23:23:00.0354768Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:03.8370437Z 2024-08-20T23:23:04.3791727Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:04.3792211Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:04.3818036Z cuda eval phlippe_resnet 2024-08-20T23:23:07.9284025Z pass 2024-08-20T23:23:07.9284812Z TIMING: backend_compile:1.48158 entire_frame_compile:3.26449 total_wall_time:3.26449 2024-08-20T23:23:07.9286144Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:6793 | ProxyTorchDispatchMode.__torch_dispatch__:1681 2024-08-20T23:23:07.9287192Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:11.6075951Z 2024-08-20T23:23:11.9579777Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:11.9580811Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:11.9582279Z cuda eval pyhpc_equation_of_state 2024-08-20T23:23:13.8662464Z pass 2024-08-20T23:23:13.8663892Z TIMING: backend_compile:1.15653 entire_frame_compile:1.70945 total_wall_time:1.70945 2024-08-20T23:23:13.8666849Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:8227 | ProxyTorchDispatchMode.__torch_dispatch__:3732 2024-08-20T23:23:13.8668064Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:17.4965479Z 2024-08-20T23:23:17.8146734Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:17.8147307Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:17.8151683Z cuda eval pyhpc_isoneutral_mixing 2024-08-20T23:23:26.8285329Z pass 2024-08-20T23:23:26.8287044Z TIMING: backend_compile:6.73099 entire_frame_compile:8.72089 total_wall_time:8.72089 2024-08-20T23:23:26.8288232Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:32445 | FakeTensor.__torch_dispatch__:18 | ProxyTorchDispatchMode.__torch_dispatch__:13834 2024-08-20T23:23:26.8289348Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:30.6175396Z 2024-08-20T23:23:31.3725277Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:23:31.4916965Z 2024-08-20T23:23:31.4917298Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:31.4918036Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:23:31.4918827Z cuda eval pyhpc_turbulent_kinetic_energy 2024-08-20T23:23:31.5371074Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:23:41.7543914Z pass 2024-08-20T23:23:41.7648510Z TIMING: backend_compile:7.71572 entire_frame_compile:9.91268 total_wall_time:9.91268 2024-08-20T23:23:41.7649886Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:34456 | FakeTensor.__torch_dispatch__:71 | ProxyTorchDispatchMode.__torch_dispatch__:14474 2024-08-20T23:23:41.7651009Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:45.5674194Z 2024-08-20T23:23:46.4011160Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:46.4011751Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:46.4024611Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-08-20T23:23:55.1962044Z pass 2024-08-20T23:23:55.1968624Z TIMING: backend_compile:1.40344 entire_frame_compile:3.11355 total_wall_time:3.11355 2024-08-20T23:23:55.1969859Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:6389 | ProxyTorchDispatchMode.__torch_dispatch__:2092 2024-08-20T23:23:55.1970845Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:58.9029271Z 2024-08-20T23:24:00.0527805Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:24:00.0528277Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:24:00.0549917Z cuda eval pytorch_stargan 2024-08-20T23:24:18.7369466Z pass 2024-08-20T23:24:18.7413364Z TIMING: backend_compile:2.30229 entire_frame_compile:3.98405 total_wall_time:3.98405 2024-08-20T23:24:18.7414937Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:10845 | FakeTensor.__torch_dispatch__:9 | ProxyTorchDispatchMode.__torch_dispatch__:3626 2024-08-20T23:24:18.7416404Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-08-20T23:24:22.4829615Z 2024-08-20T23:24:23.1965689Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:24:23.1966198Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:24:23.1988637Z cuda eval pytorch_unet 2024-08-20T23:24:34.6443889Z pass 2024-08-20T23:24:34.6637076Z TIMING: backend_compile:1.90542 entire_frame_compile:3.93823 total_wall_time:3.93823 2024-08-20T23:24:34.6638356Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:9567 | ProxyTorchDispatchMode.__torch_dispatch__:2964 2024-08-20T23:24:34.6639309Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:24:38.3246981Z 2024-08-20T23:24:39.1358363Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-08-20T23:24:39.1541254Z 2024-08-20T23:24:39.1541439Z 2024-08-20T23:24:39.2544073Z 0% 0.00/230M [00:00 2024-08-20T23:32:15.0915415Z torchbench_main() 2024-08-20T23:32:15.0916222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torchbench_main 2024-08-20T23:32:15.0917151Z main(TorchBenchmarkRunner(), original_dir) 2024-08-20T23:32:15.0918060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4087, in main 2024-08-20T23:32:15.0918824Z process_entry(0, runner, original_dir, args) 2024-08-20T23:32:15.0919620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4019, in process_entry 2024-08-20T23:32:15.0922809Z return run(runner, args, original_dir) 2024-08-20T23:32:15.0923503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4661, in run 2024-08-20T23:32:15.0928376Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-08-20T23:32:15.0929135Z AssertionError: nothing in example_inputs had a dim with 4 2024-08-20T23:32:18.6679773Z Run failed with return code: 1 2024-08-20T23:32:18.6680347Z Output: None 2024-08-20T23:32:18.6680690Z Error: None 2024-08-20T23:32:21.4879375Z 2024-08-20T23:32:21.6687654Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth 2024-08-20T23:32:21.6877561Z 2024-08-20T23:32:21.6885064Z 2024-08-20T23:32:21.7131038Z 0% 0.00/8.79M [00:00 2024-08-20T23:36:13.6799897Z W0820 23:36:13.676000 38292 torch/_dynamo/variables/tensor.py:776] [13/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:36:13.6801671Z W0820 23:36:13.676000 38292 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-20T23:36:13.6803243Z W0820 23:36:13.676000 38292 torch/_dynamo/variables/tensor.py:776] [13/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-20T23:36:13.6804300Z W0820 23:36:13.676000 38292 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:36:13.6805027Z W0820 23:36:13.676000 38292 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:36:41.3660211Z pass 2024-08-20T23:36:41.3756127Z TIMING: entire_frame_compile:46.04786 backend_compile:10.19605 total_wall_time:46.04786 2024-08-20T23:36:41.3757703Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:51797 | FakeTensor.__torch_dispatch__:42 | ProxyTorchDispatchMode.__torch_dispatch__:17420 | attempt fast:88 | fast is_contiguous:52 | slow both tensors nontrivially broadcast:36 2024-08-20T23:36:41.3759296Z Dynamo produced 25 graphs covering 1358 ops with 16 graph breaks (2 unique) 2024-08-20T23:36:45.6353928Z 2024-08-20T23:36:47.8618200Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:36:47.8618677Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:36:47.8713217Z cuda eval yolov3 2024-08-20T23:37:08.5704699Z pass 2024-08-20T23:37:08.5819506Z TIMING: backend_compile:6.57153 entire_frame_compile:14.05353 total_wall_time:14.05353 2024-08-20T23:37:08.5820688Z STATS: call_* op count: 649 | FakeTensorMode.__torch_dispatch__:26684 | FakeTensor.__torch_dispatch__:35 | ProxyTorchDispatchMode.__torch_dispatch__:6718 2024-08-20T23:37:08.5823187Z Dynamo produced 1 graphs covering 649 ops with 0 graph breaks (0 unique) 2024-08-20T23:37:09.7492640Z accuracy pass_rate=82.22% 2024-08-20T23:37:09.7493169Z calls_captured gmean=0.00x mean=526.911x 2024-08-20T23:37:09.7495801Z unique_graphs gmean=0.00x mean=1.667x 2024-08-20T23:37:09.7499532Z graph_breaks gmean=0.00x mean=0.733x 2024-08-20T23:37:09.7502875Z unique_graph_breaks gmean=0.00x mean=0.178x 2024-08-20T23:37:09.7505873Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:37:09.7509349Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:37:09.7512772Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T23:37:10.4991982Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_inference.csv 2024-08-20T23:37:10.8399478Z llama PASS 2024-08-20T23:37:10.8404061Z llama_v2_7b_16h XFAIL 2024-08-20T23:37:10.8408740Z llava XFAIL 2024-08-20T23:37:10.8413525Z maml XFAIL 2024-08-20T23:37:10.8418160Z maml_omniglot PASS 2024-08-20T23:37:10.8422785Z mnasnet1_0 PASS 2024-08-20T23:37:10.8427569Z mobilenet_v2 PASS 2024-08-20T23:37:10.8432565Z mobilenet_v2_quantized_qat XFAIL 2024-08-20T23:37:10.8437539Z mobilenet_v3_large PASS 2024-08-20T23:37:10.8443203Z moco PASS 2024-08-20T23:37:10.8447942Z moondream XFAIL 2024-08-20T23:37:10.8452646Z nanogpt PASS 2024-08-20T23:37:10.8457300Z nvidia_deeprecommender PASS 2024-08-20T23:37:10.8462070Z opacus_cifar10 PASS 2024-08-20T23:37:10.8466646Z phlippe_densenet PASS 2024-08-20T23:37:10.8471654Z phlippe_resnet PASS 2024-08-20T23:37:10.8476304Z pyhpc_equation_of_state PASS 2024-08-20T23:37:10.8481064Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:37:10.8485765Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:37:10.8490456Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:37:10.8495112Z pytorch_stargan PASS 2024-08-20T23:37:10.8499861Z pytorch_unet PASS 2024-08-20T23:37:10.8504552Z resnet152 PASS 2024-08-20T23:37:10.8509422Z resnet18 PASS 2024-08-20T23:37:10.8514193Z resnet50 PASS 2024-08-20T23:37:10.8518957Z resnet50_quantized_qat XFAIL 2024-08-20T23:37:10.8523628Z resnext50_32x4d PASS 2024-08-20T23:37:10.8528394Z sam PASS 2024-08-20T23:37:10.8533103Z shufflenet_v2_x1_0 PASS 2024-08-20T23:37:10.8537869Z soft_actor_critic PASS 2024-08-20T23:37:10.8543021Z speech_transformer PASS 2024-08-20T23:37:10.8547665Z squeezenet1_1 PASS 2024-08-20T23:37:10.8552612Z stable_diffusion_text_encoder PASS 2024-08-20T23:37:10.8557220Z stable_diffusion_unet XFAIL 2024-08-20T23:37:10.8561908Z timm_efficientnet PASS 2024-08-20T23:37:10.8566680Z timm_regnet PASS 2024-08-20T23:37:10.8571349Z timm_resnest PASS 2024-08-20T23:37:10.8576015Z timm_vision_transformer PASS 2024-08-20T23:37:10.8580802Z timm_vision_transformer_large XFAIL 2024-08-20T23:37:10.8585519Z timm_vovnet PASS 2024-08-20T23:37:10.8590518Z torch_multimodal_clip PASS 2024-08-20T23:37:10.8595140Z tts_angular PASS 2024-08-20T23:37:10.8599751Z vgg16 PASS 2024-08-20T23:37:10.8604542Z vision_maskrcnn PASS 2024-08-20T23:37:10.8609165Z yolov3 PASS 2024-08-20T23:37:10.9132516Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_inference.csv 2024-08-20T23:37:11.2489720Z llama PASS 2024-08-20T23:37:11.2492682Z llama_v2_7b_16h PASS 2024-08-20T23:37:11.2497496Z llava PASS 2024-08-20T23:37:11.2502040Z maml PASS 2024-08-20T23:37:11.2506743Z maml_omniglot PASS 2024-08-20T23:37:11.2511892Z mnasnet1_0 PASS 2024-08-20T23:37:11.2516587Z mobilenet_v2 PASS 2024-08-20T23:37:11.2521518Z mobilenet_v2_quantized_qat PASS 2024-08-20T23:37:11.2526227Z mobilenet_v3_large PASS 2024-08-20T23:37:11.2530877Z moco PASS 2024-08-20T23:37:11.2535605Z moondream PASS 2024-08-20T23:37:11.2540718Z nanogpt PASS 2024-08-20T23:37:11.2545685Z nvidia_deeprecommender PASS 2024-08-20T23:37:11.2550503Z opacus_cifar10 PASS 2024-08-20T23:37:11.2555221Z phlippe_densenet PASS 2024-08-20T23:37:11.2560035Z phlippe_resnet PASS 2024-08-20T23:37:11.2564794Z pyhpc_equation_of_state PASS 2024-08-20T23:37:11.2569505Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:37:11.2574052Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:37:11.2578894Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:37:11.2583567Z pytorch_stargan PASS 2024-08-20T23:37:11.2588539Z pytorch_unet PASS 2024-08-20T23:37:11.2593112Z resnet152 PASS 2024-08-20T23:37:11.2597998Z resnet18 PASS 2024-08-20T23:37:11.2602464Z resnet50 PASS 2024-08-20T23:37:11.2607307Z resnet50_quantized_qat PASS 2024-08-20T23:37:11.2611856Z resnext50_32x4d PASS 2024-08-20T23:37:11.2616616Z sam PASS 2024-08-20T23:37:11.2621315Z shufflenet_v2_x1_0 PASS 2024-08-20T23:37:11.2626122Z soft_actor_critic PASS 2024-08-20T23:37:11.2630918Z speech_transformer PASS 2024-08-20T23:37:11.2635515Z squeezenet1_1 PASS 2024-08-20T23:37:11.2640639Z stable_diffusion_text_encoder PASS 2024-08-20T23:37:11.2645409Z stable_diffusion_unet PASS 2024-08-20T23:37:11.2650091Z timm_efficientnet PASS 2024-08-20T23:37:11.2654781Z timm_regnet PASS 2024-08-20T23:37:11.2659278Z timm_resnest PASS 2024-08-20T23:37:11.2664105Z timm_vision_transformer PASS 2024-08-20T23:37:11.2668779Z timm_vision_transformer_large PASS 2024-08-20T23:37:11.2673613Z timm_vovnet PASS 2024-08-20T23:37:11.2678146Z torch_multimodal_clip PASS 2024-08-20T23:37:11.2683007Z tts_angular PASS 2024-08-20T23:37:11.2687494Z vgg16 PASS 2024-08-20T23:37:11.2692336Z vision_maskrcnn PASS 2024-08-20T23:37:11.2697016Z yolov3 PASS 2024-08-20T23:37:11.3219913Z + test_single_dynamo_benchmark training torchbench 1 --training --amp 2024-08-20T23:37:11.3223117Z ++ pwd 2024-08-20T23:37:11.3226742Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:37:11.3227564Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:37:11.3252127Z + local name=training 2024-08-20T23:37:11.3252575Z + shift 2024-08-20T23:37:11.3252902Z + local suite=torchbench 2024-08-20T23:37:11.3253371Z + shift 2024-08-20T23:37:11.3253656Z + local shard_id=1 2024-08-20T23:37:11.3253959Z + shift 2024-08-20T23:37:11.3254234Z + partition_flags=() 2024-08-20T23:37:11.3257910Z + local partition_flags 2024-08-20T23:37:11.3258423Z + [[ -n 2 ]] 2024-08-20T23:37:11.3258894Z + [[ -n 1 ]] 2024-08-20T23:37:11.3259517Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:37:11.3260254Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]] 2024-08-20T23:37:11.3260823Z + [[ dynamic_aot_eager_torchbench == *perf* ]] 2024-08-20T23:37:11.3261351Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]] 2024-08-20T23:37:11.3261876Z + [[ dynamic_aot_eager_torchbench == *_avx2* ]] 2024-08-20T23:37:11.3262371Z + [[ dynamic_aot_eager_torchbench == *_avx512* ]] 2024-08-20T23:37:11.3264185Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2024-08-20T23:37:17.0665965Z 2024-08-20T23:37:17.3692126Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:17.3692682Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:17.3693255Z cuda train llava 2024-08-20T23:37:17.3701865Z Traceback (most recent call last): 2024-08-20T23:37:17.3702873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:37:17.3703531Z ) = runner.load_model( 2024-08-20T23:37:17.3704480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model 2024-08-20T23:37:17.3705449Z benchmark = benchmark_cls( 2024-08-20T23:37:17.3706339Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:37:17.3707222Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:37:17.3708091Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:37:17.3709415Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:37:17.3710997Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:37:17.3712217Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:37:17.3713175Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:37:17.3713927Z self._skip_by_device_name() 2024-08-20T23:37:17.3714730Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:37:17.3715944Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:37:17.3717018Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:37:17.3717587Z 2024-08-20T23:37:17.3717707Z model_fail_to_load 2024-08-20T23:37:20.8364917Z 2024-08-20T23:37:21.4675687Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:21.4676168Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:21.4676609Z cuda train maml_omniglot 2024-08-20T23:37:24.4921835Z W0820 23:37:24.491000 38592 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:37:25.7532282Z pass 2024-08-20T23:37:25.7533019Z TIMING: entire_frame_compile:3.95067 backend_compile:2.50968 total_wall_time:3.95067 2024-08-20T23:37:25.7534405Z STATS: call_* op count: 77 | FakeTensorMode.__torch_dispatch__:3296 | attempt fast:65 | fast is_contiguous:65 | ProxyTorchDispatchMode.__torch_dispatch__:1158 | FakeTensor.__torch_dispatch__:296 2024-08-20T23:37:25.7535695Z Dynamo produced 3 graphs covering 77 ops with 7 graph breaks (5 unique) 2024-08-20T23:37:29.6107460Z 2024-08-20T23:37:32.1324758Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:32.1325229Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:37:32.1325662Z cuda train mnasnet1_0 2024-08-20T23:37:55.1016311Z W0820 23:37:55.100000 38691 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:38:11.4271986Z pass 2024-08-20T23:38:11.4315118Z TIMING: entire_frame_compile:37.3039 backend_compile:20.45181 total_wall_time:37.3039 2024-08-20T23:38:11.4316536Z STATS: call_* op count: 647 | FakeTensorMode.__torch_dispatch__:37853 | attempt fast:1046 | fast is_contiguous:1046 | ProxyTorchDispatchMode.__torch_dispatch__:12573 | FakeTensor.__torch_dispatch__:3072 2024-08-20T23:38:11.4318745Z Dynamo produced 3 graphs covering 647 ops with 7 graph breaks (5 unique) 2024-08-20T23:38:16.4740167Z 2024-08-20T23:38:18.4834093Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:38:18.4834568Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:38:18.4835017Z cuda train mobilenet_v2 2024-08-20T23:38:50.5834000Z pass 2024-08-20T23:38:50.5882860Z TIMING: entire_frame_compile:30.24828 backend_compile:17.41339 total_wall_time:30.24828 2024-08-20T23:38:50.5884256Z STATS: call_* op count: 157 | FakeTensorMode.__torch_dispatch__:27235 | attempt fast:1046 | fast is_contiguous:1046 | FakeTensor.__torch_dispatch__:162 | ProxyTorchDispatchMode.__torch_dispatch__:5206 2024-08-20T23:38:50.5885562Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-08-20T23:38:55.7278130Z 2024-08-20T23:38:57.6713410Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:38:57.6714007Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:38:57.6714582Z cuda train mobilenet_v2_quantized_qat 2024-08-20T23:38:57.6715090Z Traceback (most recent call last): 2024-08-20T23:38:57.6715814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:38:57.6716579Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:38:57.6717656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-20T23:38:57.6718677Z pred = mod(*cloned_inputs) 2024-08-20T23:38:57.6719992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-20T23:38:57.6720981Z return self._wrapped_call(self, *args, **kwargs) 2024-08-20T23:38:57.6721918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-20T23:38:57.6722792Z raise e 2024-08-20T23:38:57.6723698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-20T23:38:57.6724642Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:38:57.6726063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:38:57.6727092Z return self._call_impl(*args, **kwargs) 2024-08-20T23:38:57.6728198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:38:57.6729030Z return forward_call(*args, **kwargs) 2024-08-20T23:38:57.6729499Z File ".3", line 207, in forward 2024-08-20T23:38:57.6730238Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-08-20T23:38:57.6731490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:38:57.6732621Z return self._call_impl(*args, **kwargs) 2024-08-20T23:38:57.6733733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:38:57.6734565Z return forward_call(*args, **kwargs) 2024-08-20T23:38:57.6735531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-20T23:38:57.6736436Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-20T23:38:57.6736986Z RuntimeError: expected scalar type Float but found Half 2024-08-20T23:38:57.6737679Z 2024-08-20T23:38:57.6737983Z The above exception was the direct cause of the following exception: 2024-08-20T23:38:57.6738423Z 2024-08-20T23:38:57.6738569Z Traceback (most recent call last): 2024-08-20T23:38:57.6739215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:38:57.6739872Z ) = runner.load_model( 2024-08-20T23:38:57.6740528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:38:57.6741275Z self.validate_model(model, example_inputs) 2024-08-20T23:38:57.6742023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:38:57.6743008Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:38:57.6743476Z RuntimeError: Eager run failed 2024-08-20T23:38:57.6743730Z 2024-08-20T23:38:57.6743847Z eager_fail_to_run 2024-08-20T23:39:01.2747109Z 2024-08-20T23:39:03.6923294Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:03.6923901Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:39:03.6924429Z cuda train mobilenet_v3_large 2024-08-20T23:39:38.1656774Z W0820 23:39:38.164000 38924 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:39:56.7292137Z pass 2024-08-20T23:39:56.7345554Z TIMING: entire_frame_compile:50.88118 backend_compile:27.89714 total_wall_time:50.88118 2024-08-20T23:39:56.7346970Z STATS: call_* op count: 730 | FakeTensorMode.__torch_dispatch__:45440 | attempt fast:1363 | fast is_contiguous:1363 | FakeTensor.__torch_dispatch__:3492 | ProxyTorchDispatchMode.__torch_dispatch__:13925 2024-08-20T23:39:56.7349020Z Dynamo produced 3 graphs covering 730 ops with 7 graph breaks (5 unique) 2024-08-20T23:40:02.3980717Z 2024-08-20T23:40:04.7065055Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:40:04.7065663Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:40:04.7066217Z cuda train moco 2024-08-20T23:40:09.3282622Z [rank0]:W0820 23:40:09.327000 39023 torch/_logging/_internal.py:1058] [3/0] Profiler function will be ignored 2024-08-20T23:40:19.7937395Z [rank0]:W0820 23:40:19.792000 39023 torch/_dynamo/backends/distributed.py:89] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:40:19.8176544Z [rank0]:W0820 23:40:19.816000 39023 torch/_dynamo/backends/distributed.py:106] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:40:19.8178514Z [rank0]:W0820 23:40:19.816000 39023 torch/_dynamo/backends/distributed.py:106] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:40:19.8179982Z [rank0]:W0820 23:40:19.816000 39023 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:40:19.8181463Z [rank0]:W0820 23:40:19.816000 39023 torch/_dynamo/backends/distributed.py:106] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:40:19.8182855Z [rank0]:W0820 23:40:19.816000 39023 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ 0 │ 480 │ 94032128 │ 2024-08-20T23:40:19.8184276Z [rank0]:W0820 23:40:19.816000 39023 torch/_dynamo/backends/distributed.py:106] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:40:43.6686220Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:40:43.6687546Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:40:43.6689156Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] or: 2024-08-20T23:40:43.6690301Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:40:43.6691722Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] to include these operations in the captured graph. 2024-08-20T23:40:43.6692762Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:40:43.6693701Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break: from user code at: 2024-08-20T23:40:43.6695976Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:40:43.6698098Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] self._dequeue_and_enqueue(k) 2024-08-20T23:40:43.6699964Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:40:43.6701764Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] return func(*args, **kwargs) 2024-08-20T23:40:43.6703371Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:40:43.6704963Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] ptr = int(self.queue_ptr) 2024-08-20T23:40:43.6705887Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:40:43.6706664Z [rank0]:W0820 23:40:43.667000 39023 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:40:59.7012417Z pass 2024-08-20T23:40:59.7379615Z TIMING: entire_frame_compile:49.99172 backend_compile:26.40766 total_wall_time:49.99172 2024-08-20T23:40:59.7381250Z STATS: call_* op count: 891 | FakeTensorMode.__torch_dispatch__:52801 | attempt fast:1979 | fast is_contiguous:1968 | slow no contiguity match:11 | FakeTensor.__torch_dispatch__:132 | ProxyTorchDispatchMode.__torch_dispatch__:13177 2024-08-20T23:40:59.7382764Z Dynamo produced 10 graphs covering 891 ops with 11 graph breaks (8 unique) 2024-08-20T23:41:01.4412275Z [rank0]:[W820 23:41:01.386739709 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:41:06.6962470Z 2024-08-20T23:41:08.6761317Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:41:08.9324300Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:41:08.9325319Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:41:08.9326024Z using fused AdamW: True 2024-08-20T23:41:09.3877972Z 2024-08-20T23:41:09.3878433Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:41:09.3878947Z cuda train nanogpt 2024-08-20T23:41:35.4767826Z W0820 23:41:35.475000 39131 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:41:48.7706419Z pass 2024-08-20T23:41:48.8797385Z TIMING: entire_frame_compile:37.50086 backend_compile:23.57567 total_wall_time:37.50086 2024-08-20T23:41:48.8798809Z STATS: call_* op count: 858 | FakeTensorMode.__torch_dispatch__:43619 | attempt fast:1715 | fast is_contiguous:1715 | FakeTensor.__torch_dispatch__:2868 | ProxyTorchDispatchMode.__torch_dispatch__:15889 2024-08-20T23:41:48.8800111Z Dynamo produced 3 graphs covering 858 ops with 7 graph breaks (5 unique) 2024-08-20T23:41:53.8625005Z 2024-08-20T23:41:56.3293653Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:41:56.3294587Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:41:56.3295087Z cuda train nvidia_deeprecommender 2024-08-20T23:41:59.8499215Z W0820 23:41:59.849000 39229 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:42:01.0913871Z pass 2024-08-20T23:42:01.1215741Z TIMING: entire_frame_compile:3.78583 backend_compile:2.36411 total_wall_time:3.78583 2024-08-20T23:42:01.1217382Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:3400 | attempt fast:150 | fast is_contiguous:150 | ProxyTorchDispatchMode.__torch_dispatch__:1070 | FakeTensor.__torch_dispatch__:272 2024-08-20T23:42:01.1219038Z Dynamo produced 3 graphs covering 70 ops with 7 graph breaks (5 unique) 2024-08-20T23:42:04.8885789Z 2024-08-20T23:42:05.8626241Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:05.8626843Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:05.8627392Z cuda train opacus_cifar10 2024-08-20T23:42:05.8643046Z Traceback (most recent call last): 2024-08-20T23:42:05.8643912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:42:05.8644793Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:42:05.8645822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-20T23:42:05.8646966Z self.grad_scaler.scale(loss).backward() 2024-08-20T23:42:05.8648312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-20T23:42:05.8649318Z torch.autograd.backward( 2024-08-20T23:42:05.8650405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-20T23:42:05.8651415Z _engine_run_backward( 2024-08-20T23:42:05.8652339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-20T23:42:05.8653498Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-20T23:42:05.8654675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 98, in __call__ 2024-08-20T23:42:05.8655483Z return self.hook(module, *args, **kwargs) 2024-08-20T23:42:05.8656563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 328, in capture_backprops_hook 2024-08-20T23:42:05.8657641Z activations, backprops = self.rearrange_grad_samples( 2024-08-20T23:42:05.8658907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 384, in rearrange_grad_samples 2024-08-20T23:42:05.8659825Z raise ValueError( 2024-08-20T23:42:05.8660664Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-08-20T23:42:05.8661349Z 2024-08-20T23:42:05.8661648Z The above exception was the direct cause of the following exception: 2024-08-20T23:42:05.8662103Z 2024-08-20T23:42:05.8662255Z Traceback (most recent call last): 2024-08-20T23:42:05.8662898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:42:05.8663535Z ) = runner.load_model( 2024-08-20T23:42:05.8664204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:42:05.8664966Z self.validate_model(model, example_inputs) 2024-08-20T23:42:05.8665718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:42:05.8666480Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:42:05.8666951Z RuntimeError: Eager run failed 2024-08-20T23:42:05.8667202Z 2024-08-20T23:42:05.8667446Z eager_fail_to_run 2024-08-20T23:42:09.4751529Z 2024-08-20T23:42:10.3055785Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:10.3056368Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:10.3056860Z cuda train phlippe_densenet 2024-08-20T23:42:37.4313726Z pass 2024-08-20T23:42:37.4316732Z TIMING: entire_frame_compile:26.40508 backend_compile:14.06811 total_wall_time:26.40508 2024-08-20T23:42:37.4318295Z STATS: call_* op count: 190 | FakeTensorMode.__torch_dispatch__:27135 | attempt fast:1020 | fast is_contiguous:996 | ProxyTorchDispatchMode.__torch_dispatch__:6202 | slow no contiguity match:24 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:42:37.4320066Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-08-20T23:42:42.4690936Z 2024-08-20T23:42:43.1926793Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:43.1927263Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:43.1927724Z cuda train phlippe_resnet 2024-08-20T23:42:52.6841673Z pass 2024-08-20T23:42:52.6845070Z TIMING: entire_frame_compile:9.06038 backend_compile:5.13108 total_wall_time:9.06038 2024-08-20T23:42:52.6846751Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:10010 | attempt fast:414 | fast is_contiguous:414 | ProxyTorchDispatchMode.__torch_dispatch__:2343 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:42:52.6848044Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-08-20T23:42:56.7575753Z 2024-08-20T23:42:58.0013897Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:58.0014496Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:42:58.0015047Z cuda train pytorch_CycleGAN_and_pix2pix 2024-08-20T23:43:11.7656429Z pass 2024-08-20T23:43:11.7747267Z TIMING: entire_frame_compile:4.84168 backend_compile:2.75319 total_wall_time:4.84168 2024-08-20T23:43:11.7748754Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:8246 | ProxyTorchDispatchMode.__torch_dispatch__:3021 | FakeTensor.__torch_dispatch__:25 2024-08-20T23:43:11.7749863Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-08-20T23:43:15.6451073Z 2024-08-20T23:43:17.1491000Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:43:17.1491546Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:43:17.1492007Z cuda train pytorch_stargan 2024-08-20T23:43:57.9914031Z pass 2024-08-20T23:43:58.0103070Z TIMING: entire_frame_compile:14.61483 backend_compile:9.73914 total_wall_time:14.61483 2024-08-20T23:43:58.0104631Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:15858 | attempt fast:611 | fast is_contiguous:611 | FakeTensor.__torch_dispatch__:53 | ProxyTorchDispatchMode.__torch_dispatch__:3747 2024-08-20T23:43:58.0105954Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T23:44:02.3754962Z 2024-08-20T23:44:03.3016682Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:44:03.3017170Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:44:03.3017602Z cuda train pytorch_unet 2024-08-20T23:44:34.0482056Z W0820 23:44:34.047000 39758 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:44:41.6625232Z pass_due_to_skip 2024-08-20T23:44:41.8363308Z TIMING: entire_frame_compile:18.20099 backend_compile:10.7548 total_wall_time:18.20099 2024-08-20T23:44:41.8365010Z STATS: call_* op count: 314 | FakeTensorMode.__torch_dispatch__:19299 | attempt fast:526 | fast is_contiguous:522 | ProxyTorchDispatchMode.__torch_dispatch__:6972 | slow no contiguity match:4 | FakeTensor.__torch_dispatch__:1428 2024-08-20T23:44:41.8366567Z Dynamo produced 3 graphs covering 314 ops with 7 graph breaks (5 unique) 2024-08-20T23:44:46.1205087Z 2024-08-20T23:44:48.5111353Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:44:48.5111967Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:44:48.5112468Z cuda train resnet152 2024-08-20T23:45:55.7802458Z W0820 23:45:55.779000 39850 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:46:44.9133907Z pass 2024-08-20T23:46:44.9816936Z TIMING: entire_frame_compile:109.59889 backend_compile:61.96218 total_wall_time:109.59889 2024-08-20T23:46:44.9818522Z STATS: call_* op count: 1937 | FakeTensorMode.__torch_dispatch__:114330 | attempt fast:3103 | fast is_contiguous:3103 | ProxyTorchDispatchMode.__torch_dispatch__:38010 | FakeTensor.__torch_dispatch__:8542 2024-08-20T23:46:44.9820378Z Dynamo produced 3 graphs covering 1937 ops with 7 graph breaks (5 unique) 2024-08-20T23:46:52.5311541Z 2024-08-20T23:46:53.7899730Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:53.7900248Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:46:53.7900696Z cuda train resnet18 2024-08-20T23:47:04.0440894Z pass 2024-08-20T23:47:04.0520765Z TIMING: entire_frame_compile:8.64146 backend_compile:5.23643 total_wall_time:8.64146 2024-08-20T23:47:04.0522312Z STATS: call_* op count: 73 | FakeTensorMode.__torch_dispatch__:9374 | attempt fast:412 | fast is_contiguous:412 | ProxyTorchDispatchMode.__torch_dispatch__:2258 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:47:04.0524032Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-08-20T23:47:08.1618480Z 2024-08-20T23:47:09.9876027Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:47:09.9876556Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:47:09.9878693Z cuda train resnet50 2024-08-20T23:47:36.3704746Z pass 2024-08-20T23:47:36.3896063Z TIMING: entire_frame_compile:21.9942 backend_compile:13.1054 total_wall_time:21.9942 2024-08-20T23:47:36.3897730Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:24230 | attempt fast:1063 | fast is_contiguous:1063 | ProxyTorchDispatchMode.__torch_dispatch__:5826 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:47:36.3899024Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-08-20T23:47:41.0699716Z 2024-08-20T23:47:43.1149644Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:47:43.1150140Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:47:43.1150664Z cuda train resnet50_quantized_qat 2024-08-20T23:47:43.1151183Z Traceback (most recent call last): 2024-08-20T23:47:43.1151947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:47:43.1152789Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:47:43.1153822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-20T23:47:43.1154760Z pred = mod(*cloned_inputs) 2024-08-20T23:47:43.1155915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-20T23:47:43.1156858Z return self._wrapped_call(self, *args, **kwargs) 2024-08-20T23:47:43.1157957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-20T23:47:43.1158793Z raise e 2024-08-20T23:47:43.1159632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-20T23:47:43.1160709Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:47:43.1161970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:47:43.1163008Z return self._call_impl(*args, **kwargs) 2024-08-20T23:47:43.1164030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:47:43.1164937Z return forward_call(*args, **kwargs) 2024-08-20T23:47:43.1165456Z File ".3", line 167, in forward 2024-08-20T23:47:43.1166112Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-08-20T23:47:43.1167413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:47:43.1168372Z return self._call_impl(*args, **kwargs) 2024-08-20T23:47:43.1169385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:47:43.1170277Z return forward_call(*args, **kwargs) 2024-08-20T23:47:43.1171367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-20T23:47:43.1172356Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-20T23:47:43.1173281Z RuntimeError: expected scalar type Float but found Half 2024-08-20T23:47:43.1173673Z 2024-08-20T23:47:43.1174031Z The above exception was the direct cause of the following exception: 2024-08-20T23:47:43.1174482Z 2024-08-20T23:47:43.1174628Z Traceback (most recent call last): 2024-08-20T23:47:43.1175347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:47:43.1176065Z ) = runner.load_model( 2024-08-20T23:47:43.1176791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:47:43.1177620Z self.validate_model(model, example_inputs) 2024-08-20T23:47:43.1178639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:47:43.1179493Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:47:43.1180021Z RuntimeError: Eager run failed 2024-08-20T23:47:43.1180294Z 2024-08-20T23:47:43.1180420Z eager_fail_to_run 2024-08-20T23:47:46.7550944Z 2024-08-20T23:47:48.7244528Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:47:48.7245002Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:47:48.7245445Z cuda train resnext50_32x4d 2024-08-20T23:48:16.6585842Z W0820 23:48:16.657000 40181 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:48:33.1492647Z pass 2024-08-20T23:48:33.1698612Z TIMING: entire_frame_compile:38.22026 backend_compile:21.45152 total_wall_time:38.22026 2024-08-20T23:48:33.1700046Z STATS: call_* op count: 679 | FakeTensorMode.__torch_dispatch__:39420 | attempt fast:1063 | fast is_contiguous:1063 | ProxyTorchDispatchMode.__torch_dispatch__:13088 | FakeTensor.__torch_dispatch__:3026 2024-08-20T23:48:33.1701370Z Dynamo produced 3 graphs covering 679 ops with 7 graph breaks (5 unique) 2024-08-20T23:48:38.1507813Z 2024-08-20T23:48:45.7375839Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:45.7376507Z loading model: 0it [00:07, ?it/s] 2024-08-20T23:48:45.7379397Z cuda train sam 2024-08-20T23:48:45.7388038Z Traceback (most recent call last): 2024-08-20T23:48:45.7389190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:48:45.7390365Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:48:45.7391676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-20T23:48:45.7392988Z self.grad_scaler.scale(loss).backward() 2024-08-20T23:48:45.7394551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-20T23:48:45.7395704Z torch.autograd.backward( 2024-08-20T23:48:45.7397046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-20T23:48:45.7398232Z _engine_run_backward( 2024-08-20T23:48:45.7399609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-20T23:48:45.7401321Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-20T23:48:45.7402719Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-08-20T23:48:45.7403486Z 2024-08-20T23:48:45.7403929Z The above exception was the direct cause of the following exception: 2024-08-20T23:48:45.7404600Z 2024-08-20T23:48:45.7404822Z Traceback (most recent call last): 2024-08-20T23:48:45.7405782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:48:45.7406794Z ) = runner.load_model( 2024-08-20T23:48:45.7407762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:48:45.7408883Z self.validate_model(model, example_inputs) 2024-08-20T23:48:45.7409991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:48:45.7411615Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:48:45.7412319Z RuntimeError: Eager run failed 2024-08-20T23:48:45.7412708Z 2024-08-20T23:48:45.7412890Z eager_fail_to_run 2024-08-20T23:48:49.3795925Z 2024-08-20T23:48:50.6963073Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:50.6963676Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:48:50.6964150Z cuda train shufflenet_v2_x1_0 2024-08-20T23:49:16.3969791Z pass 2024-08-20T23:49:16.3979539Z TIMING: entire_frame_compile:24.67499 backend_compile:14.50914 total_wall_time:24.67499 2024-08-20T23:49:16.3981287Z STATS: call_* op count: 275 | FakeTensorMode.__torch_dispatch__:27830 | attempt fast:1075 | fast is_contiguous:1075 | ProxyTorchDispatchMode.__torch_dispatch__:6673 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:49:16.3982609Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique) 2024-08-20T23:49:21.3249076Z 2024-08-20T23:49:22.1686878Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:49:22.1687363Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:49:22.1687809Z cuda train soft_actor_critic 2024-08-20T23:49:24.5362852Z W0820 23:49:24.535000 40436 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:49:25.1246291Z pass 2024-08-20T23:49:25.1246958Z TIMING: entire_frame_compile:2.69483 backend_compile:1.67026 total_wall_time:2.69483 2024-08-20T23:49:25.1248542Z STATS: call_* op count: 56 | FakeTensorMode.__torch_dispatch__:2060 | attempt fast:46 | fast is_contiguous:46 | ProxyTorchDispatchMode.__torch_dispatch__:686 | FakeTensor.__torch_dispatch__:156 2024-08-20T23:49:25.1249980Z Dynamo produced 4 graphs covering 56 ops with 6 graph breaks (5 unique) 2024-08-20T23:49:28.9416671Z 2024-08-20T23:49:30.2132531Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:49:30.2133274Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:49:30.2133924Z Traceback (most recent call last): 2024-08-20T23:49:30.2135030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in 2024-08-20T23:49:30.2136126Z torchbench_main() 2024-08-20T23:49:30.2137541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torchbench_main 2024-08-20T23:49:30.2138817Z main(TorchBenchmarkRunner(), original_dir) 2024-08-20T23:49:30.2139916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4087, in main 2024-08-20T23:49:30.2142663Z process_entry(0, runner, original_dir, args) 2024-08-20T23:49:30.2143843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4019, in process_entry 2024-08-20T23:49:30.2149075Z return run(runner, args, original_dir) 2024-08-20T23:49:30.2150159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4661, in run 2024-08-20T23:49:30.2156387Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-08-20T23:49:30.2157417Z AssertionError: nothing in example_inputs had a dim with 32 2024-08-20T23:49:30.9459138Z Run failed with return code: 1 2024-08-20T23:49:30.9459595Z Output: None 2024-08-20T23:49:30.9459898Z Error: None 2024-08-20T23:49:33.7643334Z 2024-08-20T23:49:34.9140123Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:49:34.9140579Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:49:34.9141028Z cuda train squeezenet1_1 2024-08-20T23:49:42.1671836Z pass 2024-08-20T23:49:42.1684523Z TIMING: entire_frame_compile:6.09451 backend_compile:4.09435 total_wall_time:6.09451 2024-08-20T23:49:42.1685893Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:5624 | ProxyTorchDispatchMode.__torch_dispatch__:1964 | attempt fast:8 | fast is_contiguous:8 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:49:42.1687157Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-08-20T23:49:46.2045136Z 2024-08-20T23:49:47.4388185Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:49:47.4389308Z 2024-08-20T23:49:47.5551972Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-08-20T23:51:34.6916356Z pass 2024-08-20T23:51:34.6967685Z TIMING: entire_frame_compile:62.59883 backend_compile:35.30513 total_wall_time:62.59883 2024-08-20T23:51:34.6970753Z STATS: call_* op count: 899 | FakeTensorMode.__torch_dispatch__:55450 | attempt fast:1536 | fast is_contiguous:1536 | FakeTensor.__torch_dispatch__:4330 | ProxyTorchDispatchMode.__torch_dispatch__:18598 2024-08-20T23:51:34.6973348Z Dynamo produced 3 graphs covering 899 ops with 7 graph breaks (5 unique) 2024-08-20T23:51:40.9888027Z 2024-08-20T23:51:43.7851187Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:51:43.7851647Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:51:43.7852081Z cuda train timm_regnet 2024-08-20T23:52:31.5223367Z pass 2024-08-20T23:52:31.5654960Z TIMING: entire_frame_compile:37.69464 backend_compile:21.50164 total_wall_time:37.69464 2024-08-20T23:52:31.5658067Z STATS: call_* op count: 323 | FakeTensorMode.__torch_dispatch__:39560 | attempt fast:1471 | fast is_contiguous:1471 | ProxyTorchDispatchMode.__torch_dispatch__:10183 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:52:31.5660667Z Dynamo produced 2 graphs covering 323 ops with 6 graph breaks (5 unique) 2024-08-20T23:52:37.1081113Z 2024-08-20T23:52:39.1113967Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:39.1114440Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:52:39.1114870Z cuda train timm_resnest 2024-08-20T23:52:57.1753044Z W0820 23:52:57.174000 41022 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:53:05.9584973Z pass 2024-08-20T23:53:05.9660379Z TIMING: entire_frame_compile:22.57153 backend_compile:12.71773 total_wall_time:22.57153 2024-08-20T23:53:05.9661892Z STATS: call_* op count: 401 | FakeTensorMode.__torch_dispatch__:21728 | attempt fast:529 | fast is_contiguous:521 | ProxyTorchDispatchMode.__torch_dispatch__:7268 | slow no contiguity match:8 | FakeTensor.__torch_dispatch__:1638 2024-08-20T23:53:05.9663336Z Dynamo produced 3 graphs covering 401 ops with 7 graph breaks (5 unique) 2024-08-20T23:53:10.6751727Z 2024-08-20T23:53:12.2313343Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:12.2314507Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:53:12.2315158Z cuda train timm_vision_transformer 2024-08-20T23:53:36.7606073Z pass 2024-08-20T23:53:36.7766615Z TIMING: entire_frame_compile:23.18407 backend_compile:14.66385 total_wall_time:23.18407 2024-08-20T23:53:36.7768022Z STATS: call_* op count: 267 | FakeTensorMode.__torch_dispatch__:26975 | attempt fast:1591 | fast is_contiguous:1591 | FakeTensor.__torch_dispatch__:166 | ProxyTorchDispatchMode.__torch_dispatch__:7504 2024-08-20T23:53:36.7769322Z Dynamo produced 2 graphs covering 267 ops with 6 graph breaks (5 unique) 2024-08-20T23:53:41.6954060Z 2024-08-20T23:53:54.3428949Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:54.3429418Z loading model: 0it [00:12, ?it/s] 2024-08-20T23:53:54.3429883Z cuda train timm_vision_transformer_large 2024-08-20T23:53:54.3432993Z pass_due_to_skip 2024-08-20T23:53:54.6116094Z TIMING: 2024-08-20T23:53:54.6116428Z STATS: call_* op count: 0 2024-08-20T23:53:54.6117086Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:53:58.3271232Z 2024-08-20T23:54:00.3669855Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:54:00.3670455Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:54:00.3671008Z cuda train timm_vovnet 2024-08-20T23:54:24.5013133Z pass 2024-08-20T23:54:24.5189218Z TIMING: entire_frame_compile:17.89368 backend_compile:10.37085 total_wall_time:17.89368 2024-08-20T23:54:24.5191019Z STATS: call_* op count: 134 | FakeTensorMode.__torch_dispatch__:19066 | attempt fast:779 | fast is_contiguous:749 | ProxyTorchDispatchMode.__torch_dispatch__:4649 | slow no contiguity match:30 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:54:24.5192786Z Dynamo produced 2 graphs covering 134 ops with 6 graph breaks (5 unique) 2024-08-20T23:54:29.1685478Z 2024-08-20T23:54:31.1807434Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:54:31.1807908Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:54:31.1808398Z cuda train torch_multimodal_clip 2024-08-20T23:55:15.2630508Z W0820 23:55:15.262000 41374 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:55:42.4753028Z pass 2024-08-20T23:55:42.6226481Z TIMING: entire_frame_compile:66.04512 backend_compile:39.516 total_wall_time:66.04512 2024-08-20T23:55:42.6228282Z STATS: call_* op count: 1404 | FakeTensorMode.__torch_dispatch__:81409 | attempt fast:1485 | fast is_contiguous:1430 | FakeTensor.__torch_dispatch__:5896 | slow no contiguity match:55 | ProxyTorchDispatchMode.__torch_dispatch__:32713 2024-08-20T23:55:42.6229746Z Dynamo produced 3 graphs covering 1404 ops with 7 graph breaks (5 unique) 2024-08-20T23:55:48.8042331Z 2024-08-20T23:55:49.4796728Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:49.4797341Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:49.4797911Z cuda train tts_angular 2024-08-20T23:55:51.0167857Z W0820 23:55:51.015000 41473 torch/_logging/_internal.py:1058] [10/0] Profiler function will be ignored 2024-08-20T23:55:52.4741403Z pass 2024-08-20T23:55:52.4844850Z TIMING: entire_frame_compile:2.50484 backend_compile:1.52021 total_wall_time:2.50484 2024-08-20T23:55:52.4846251Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:1768 | FakeTensor.__torch_dispatch__:334 | ProxyTorchDispatchMode.__torch_dispatch__:825 2024-08-20T23:55:52.4847845Z Dynamo produced 5 graphs covering 70 ops with 9 graph breaks (6 unique) 2024-08-20T23:55:56.2815070Z 2024-08-20T23:55:58.9068131Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:58.9068611Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:55:58.9071993Z cuda train vgg16 2024-08-20T23:56:20.3267162Z pass 2024-08-20T23:56:20.3509369Z TIMING: entire_frame_compile:4.01268 backend_compile:2.82015 total_wall_time:4.01268 2024-08-20T23:56:20.3511173Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:3596 | attempt fast:24 | fast is_contiguous:24 | ProxyTorchDispatchMode.__torch_dispatch__:1238 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:56:20.3513622Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-08-20T23:56:24.2771475Z 2024-08-20T23:56:27.6242709Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:56:27.6243224Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:56:27.6243672Z cuda train vision_maskrcnn 2024-08-20T23:56:54.0624406Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:56:54.0625859Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:56:54.0627000Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] or: 2024-08-20T23:56:54.0628136Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:56:54.0629350Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] to include these operations in the captured graph. 2024-08-20T23:56:54.0630323Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-20T23:56:54.0631270Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break: from user code at: 2024-08-20T23:56:54.0633683Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104 2024-08-20T23:56:54.0635687Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets) 2024-08-20T23:56:54.0637857Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward 2024-08-20T23:56:54.0639522Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] box_features = self.box_roi_pool(features, proposals, image_shapes) 2024-08-20T23:56:54.0641255Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward 2024-08-20T23:56:54.0642754Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] self.scales, self.map_levels = _setup_scales( 2024-08-20T23:56:54.0644436Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales 2024-08-20T23:56:54.0646061Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:56:54.0647866Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in 2024-08-20T23:56:54.0649466Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:56:54.0651602Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-20T23:56:54.0653181Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-20T23:56:54.0654193Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-20T23:56:54.0654928Z W0820 23:56:54.061000 41671 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-20T23:57:49.5384531Z W0820 23:57:49.537000 41671 torch/_dynamo/convert_frame.py:831] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-08-20T23:57:49.5387235Z W0820 23:57:49.537000 41671 torch/_dynamo/convert_frame.py:831] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-08-20T23:57:49.5389209Z W0820 23:57:49.537000 41671 torch/_dynamo/convert_frame.py:831] [30/8] last reason: 30/0: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 619 2024-08-20T23:57:49.5390625Z W0820 23:57:49.537000 41671 torch/_dynamo/convert_frame.py:831] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-08-20T23:57:49.5392164Z W0820 23:57:49.537000 41671 torch/_dynamo/convert_frame.py:831] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-08-20T23:58:19.7862119Z pass 2024-08-20T23:58:19.8521499Z TIMING: entire_frame_compile:103.59569 backend_compile:56.73907 total_wall_time:103.59569 2024-08-20T23:58:19.8523751Z STATS: call_* op count: 2436 | FakeTensorMode.__torch_dispatch__:115903 | FakeTensor.__torch_dispatch__:680 | ProxyTorchDispatchMode.__torch_dispatch__:41391 | attempt fast:1737 | slow no contiguity match:182 | fast is_contiguous:1219 | slow both tensors nontrivially broadcast:336 2024-08-20T23:58:19.8525471Z Dynamo produced 51 graphs covering 2436 ops with 33 graph breaks (8 unique) 2024-08-20T23:58:26.4257753Z 2024-08-20T23:58:29.2970867Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:58:29.2971492Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:58:29.2971997Z cuda train yolov3 2024-08-20T23:59:31.2667200Z W0820 23:59:31.265000 41770 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:00:37.5751460Z pass 2024-08-21T00:00:37.6372273Z TIMING: entire_frame_compile:109.46523 backend_compile:63.8175 total_wall_time:109.46523 2024-08-21T00:00:37.6373819Z STATS: call_* op count: 1377 | FakeTensorMode.__torch_dispatch__:104863 | attempt fast:4332 | fast is_contiguous:4326 | FakeTensor.__torch_dispatch__:4758 | ProxyTorchDispatchMode.__torch_dispatch__:28065 | slow no contiguity match:6 2024-08-21T00:00:37.6375264Z Dynamo produced 4 graphs covering 1377 ops with 8 graph breaks (5 unique) 2024-08-21T00:00:43.3385938Z accuracy pass_rate=78.38% 2024-08-21T00:00:43.3389468Z calls_captured gmean=0.00x mean=422.054x 2024-08-21T00:00:43.3393533Z unique_graphs gmean=0.00x mean=3.649x 2024-08-21T00:00:43.3397097Z graph_breaks gmean=0.00x mean=6.135x 2024-08-21T00:00:43.3400883Z unique_graph_breaks gmean=0.00x mean=4.243x 2024-08-21T00:00:43.3404737Z autograd_captures gmean=0.00x mean=0.000x 2024-08-21T00:00:43.3408421Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-21T00:00:43.3412245Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-21T00:00:43.9355674Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_training.csv 2024-08-21T00:00:44.2713696Z llava XFAIL 2024-08-21T00:00:44.2718168Z maml_omniglot PASS 2024-08-21T00:00:44.2723270Z mnasnet1_0 PASS 2024-08-21T00:00:44.2728450Z mobilenet_v2 PASS 2024-08-21T00:00:44.2733532Z mobilenet_v2_quantized_qat XFAIL 2024-08-21T00:00:44.2738894Z mobilenet_v3_large PASS 2024-08-21T00:00:44.2744694Z moco PASS 2024-08-21T00:00:44.2749633Z nanogpt PASS 2024-08-21T00:00:44.2754678Z nvidia_deeprecommender PASS 2024-08-21T00:00:44.2759588Z opacus_cifar10 XFAIL 2024-08-21T00:00:44.2764572Z phlippe_densenet PASS 2024-08-21T00:00:44.2769518Z phlippe_resnet PASS 2024-08-21T00:00:44.2774476Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-21T00:00:44.2779655Z pytorch_stargan PASS 2024-08-21T00:00:44.2784287Z pytorch_unet XFAIL 2024-08-21T00:00:44.2789508Z resnet152 PASS 2024-08-21T00:00:44.2794500Z resnet18 PASS 2024-08-21T00:00:44.2799610Z resnet50 PASS 2024-08-21T00:00:44.2804542Z resnet50_quantized_qat XFAIL 2024-08-21T00:00:44.2809501Z resnext50_32x4d PASS 2024-08-21T00:00:44.2814525Z sam XFAIL 2024-08-21T00:00:44.2819597Z shufflenet_v2_x1_0 PASS 2024-08-21T00:00:44.2824507Z soft_actor_critic PASS 2024-08-21T00:00:44.2829642Z squeezenet1_1 PASS 2024-08-21T00:00:44.2834516Z stable_diffusion_text_encoder PASS 2024-08-21T00:00:44.2839957Z stable_diffusion_unet XFAIL 2024-08-21T00:00:44.2845273Z timm_efficientnet PASS 2024-08-21T00:00:44.2850172Z timm_regnet PASS 2024-08-21T00:00:44.2855109Z timm_resnest PASS 2024-08-21T00:00:44.2860082Z timm_vision_transformer PASS 2024-08-21T00:00:44.2865033Z timm_vision_transformer_large XFAIL 2024-08-21T00:00:44.2870297Z timm_vovnet PASS 2024-08-21T00:00:44.2875179Z torch_multimodal_clip PASS 2024-08-21T00:00:44.2880140Z tts_angular PASS 2024-08-21T00:00:44.2885067Z vgg16 PASS 2024-08-21T00:00:44.2890033Z vision_maskrcnn PASS 2024-08-21T00:00:44.2895166Z yolov3 PASS_BUT_FLAKY: accuracy=pass, expected=fail_accuracy 2024-08-21T00:00:44.3413700Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_training.csv 2024-08-21T00:00:44.6765303Z llava PASS 2024-08-21T00:00:44.6769140Z maml_omniglot PASS 2024-08-21T00:00:44.6773520Z mnasnet1_0 PASS 2024-08-21T00:00:44.6777986Z mobilenet_v2 PASS 2024-08-21T00:00:44.6782409Z mobilenet_v2_quantized_qat PASS 2024-08-21T00:00:44.6786977Z mobilenet_v3_large PASS 2024-08-21T00:00:44.6791844Z moco PASS 2024-08-21T00:00:44.6796039Z nanogpt PASS 2024-08-21T00:00:44.6800293Z nvidia_deeprecommender PASS 2024-08-21T00:00:44.6804919Z opacus_cifar10 PASS 2024-08-21T00:00:44.6809293Z phlippe_densenet PASS 2024-08-21T00:00:44.6813599Z phlippe_resnet PASS 2024-08-21T00:00:44.6817944Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-21T00:00:44.6822337Z pytorch_stargan PASS 2024-08-21T00:00:44.6826805Z pytorch_unet PASS 2024-08-21T00:00:44.6831279Z resnet152 PASS 2024-08-21T00:00:44.6835668Z resnet18 PASS 2024-08-21T00:00:44.6840731Z resnet50 PASS 2024-08-21T00:00:44.6845045Z resnet50_quantized_qat PASS 2024-08-21T00:00:44.6849458Z resnext50_32x4d PASS 2024-08-21T00:00:44.6853648Z sam PASS 2024-08-21T00:00:44.6858151Z shufflenet_v2_x1_0 PASS 2024-08-21T00:00:44.6862369Z soft_actor_critic PASS 2024-08-21T00:00:44.6866730Z squeezenet1_1 PASS 2024-08-21T00:00:44.6871285Z stable_diffusion_text_encoder PASS 2024-08-21T00:00:44.6875586Z stable_diffusion_unet PASS 2024-08-21T00:00:44.6879856Z timm_efficientnet PASS 2024-08-21T00:00:44.6884359Z timm_regnet PASS 2024-08-21T00:00:44.6888612Z timm_resnest PASS 2024-08-21T00:00:44.6893102Z timm_vision_transformer PASS 2024-08-21T00:00:44.6897629Z timm_vision_transformer_large PASS 2024-08-21T00:00:44.6901826Z timm_vovnet PASS 2024-08-21T00:00:44.6906087Z torch_multimodal_clip PASS 2024-08-21T00:00:44.6910709Z tts_angular PASS 2024-08-21T00:00:44.6914930Z vgg16 PASS 2024-08-21T00:00:44.6919429Z vision_maskrcnn PASS 2024-08-21T00:00:44.6923659Z yolov3 PASS 2024-08-21T00:00:44.7438911Z + cleanup_workspace 2024-08-21T00:00:44.7439933Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-21T00:00:44.7442458Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-21T00:00:44.7443722Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-21T00:00:44.7444358Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-21T00:00:44.7445190Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-21T00:00:44.7446145Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-21T00:00:44.7446848Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-21T00:00:45.5314356Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-21T00:00:45.5314886Z cat test/**/*_toprint.log || true 2024-08-21T00:00:45.5329091Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:45.5329601Z env: 2024-08-21T00:00:45.5329886Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:45.5330342Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:45.5331076Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:45.5331716Z ##[endgroup] 2024-08-21T00:00:45.5427220Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-21T00:00:45.5460528Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:00:45.5461004Z kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:00:45.5469303Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:45.5469802Z env: 2024-08-21T00:00:45.5470079Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:45.5470538Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:45.5471276Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:45.5471940Z MONITOR_SCRIPT_PID: 127411 2024-08-21T00:00:45.5472289Z ##[endgroup] 2024-08-21T00:00:45.5665855Z Prepare all required actions 2024-08-21T00:00:45.5666312Z Getting action download info 2024-08-21T00:00:45.7553625Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-21T00:00:45.9388364Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-21T00:00:45.9388839Z with: 2024-08-21T00:00:45.9389413Z file-suffix: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057 2024-08-21T00:00:45.9390114Z s3-bucket: gha-artifacts 2024-08-21T00:00:45.9390450Z env: 2024-08-21T00:00:45.9390733Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:45.9391190Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:45.9391917Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:45.9392736Z ##[endgroup] 2024-08-21T00:00:45.9419339Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-21T00:00:45.9419991Z # Remove any previous test jsons if they exist 2024-08-21T00:00:45.9420505Z rm -f test-jsons-*.zip 2024-08-21T00:00:45.9421028Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-21T00:00:45.9430221Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:45.9430727Z env: 2024-08-21T00:00:45.9431010Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:45.9431463Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:45.9432197Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:45.9433187Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057 2024-08-21T00:00:45.9433871Z ##[endgroup] 2024-08-21T00:00:45.9832495Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-21T00:00:45.9862879Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-21T00:00:45.9863568Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-21T00:00:45.9869014Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-21T00:00:45.9874621Z adding: test/slow_tests.json (deflated 82%) 2024-08-21T00:00:45.9906444Z ##[group]Run # Remove any previous test reports if they exist 2024-08-21T00:00:45.9907075Z # Remove any previous test reports if they exist 2024-08-21T00:00:45.9907710Z rm -f test-reports-*.zip 2024-08-21T00:00:45.9908293Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-21T00:00:45.9917229Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:45.9917717Z env: 2024-08-21T00:00:45.9918002Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:45.9918466Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:45.9919195Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:45.9920124Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057 2024-08-21T00:00:45.9920822Z ##[endgroup] 2024-08-21T00:00:46.0180015Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-08-21T00:00:46.0181517Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-08-21T00:00:46.0183051Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-21T00:00:46.0184146Z adding: test/test-reports/training_torchbench.csv (deflated 67%) 2024-08-21T00:00:46.0190741Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-08-21T00:00:46.0191994Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-21T00:00:46.0224720Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-21T00:00:46.0225362Z # Remove any previous usage logs if they exist 2024-08-21T00:00:46.0225884Z rm -f logs-*.zip 2024-08-21T00:00:46.0226747Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-21T00:00:46.0227642Z # so check to see if the file exists first 2024-08-21T00:00:46.0228145Z if [ -f 'usage_log.txt' ]; then 2024-08-21T00:00:46.0228679Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-21T00:00:46.0229168Z fi 2024-08-21T00:00:46.0229541Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-21T00:00:46.0230112Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-21T00:00:46.0230607Z fi 2024-08-21T00:00:46.0240006Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:46.0240510Z env: 2024-08-21T00:00:46.0240784Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:46.0241241Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:46.0242142Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:46.0243072Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057 2024-08-21T00:00:46.0243749Z ##[endgroup] 2024-08-21T00:00:46.0348889Z adding: usage_log.txt (deflated 92%) 2024-08-21T00:00:46.0441738Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-21T00:00:46.0442608Z # Remove any previous debugging artifacts if they exist 2024-08-21T00:00:46.0443270Z rm -f debug-*.zip 2024-08-21T00:00:46.0443740Z if [ -d 'test/debug' ]; then 2024-08-21T00:00:46.0444342Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-21T00:00:46.0444914Z fi 2024-08-21T00:00:46.0454277Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:46.0454874Z env: 2024-08-21T00:00:46.0455216Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:46.0455763Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:46.0456644Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:46.0457752Z FILE_SUFFIX: test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057 2024-08-21T00:00:46.0458554Z ##[endgroup] 2024-08-21T00:00:46.0559526Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:00:46.0559957Z with: 2024-08-21T00:00:46.0560241Z s3-bucket: gha-artifacts 2024-08-21T00:00:46.0560669Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-21T00:00:46.0561142Z retention-days: 14 2024-08-21T00:00:46.0561469Z if-no-files-found: warn 2024-08-21T00:00:46.0561823Z path: test-jsons-*.zip 2024-08-21T00:00:46.0562159Z name: artifact 2024-08-21T00:00:46.0562455Z region: us-east-1 2024-08-21T00:00:46.0562753Z env: 2024-08-21T00:00:46.0563027Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:46.0563522Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:46.0564238Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:46.0564873Z ##[endgroup] 2024-08-21T00:00:46.4077104Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:00:46.4077910Z With the provided path, there will be 1 file uploaded 2024-08-21T00:00:46.4078680Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-21T00:00:46.8103020Z Starting upload of test-jsons-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057.zip 2024-08-21T00:00:46.9339752Z Finished upload of test-jsons-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057.zip 2024-08-21T00:00:46.9620675Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:00:46.9621128Z with: 2024-08-21T00:00:46.9621411Z s3-bucket: gha-artifacts 2024-08-21T00:00:46.9621849Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-21T00:00:46.9622319Z retention-days: 14 2024-08-21T00:00:46.9622662Z if-no-files-found: error 2024-08-21T00:00:46.9623036Z path: test-reports-*.zip 2024-08-21T00:00:46.9623390Z name: artifact 2024-08-21T00:00:46.9623698Z region: us-east-1 2024-08-21T00:00:46.9624052Z env: 2024-08-21T00:00:46.9624538Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:46.9625002Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:46.9625729Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:46.9626365Z ##[endgroup] 2024-08-21T00:00:47.3175854Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:00:47.3177107Z With the provided path, there will be 1 file uploaded 2024-08-21T00:00:47.3178326Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-21T00:00:47.3232154Z Starting upload of test-reports-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057.zip 2024-08-21T00:00:47.4489133Z Finished upload of test-reports-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057.zip 2024-08-21T00:00:47.4769723Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:00:47.4770166Z with: 2024-08-21T00:00:47.4770458Z s3-bucket: gha-artifacts 2024-08-21T00:00:47.4770910Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-21T00:00:47.4771387Z retention-days: 14 2024-08-21T00:00:47.4771724Z if-no-files-found: ignore 2024-08-21T00:00:47.4772087Z path: logs-*.zip 2024-08-21T00:00:47.4772401Z name: artifact 2024-08-21T00:00:47.4772706Z region: us-east-1 2024-08-21T00:00:47.4773007Z env: 2024-08-21T00:00:47.4773288Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:47.4773744Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:47.4774471Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:47.4775100Z ##[endgroup] 2024-08-21T00:00:47.8362419Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:00:47.8363089Z With the provided path, there will be 1 file uploaded 2024-08-21T00:00:47.8364334Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-21T00:00:47.8420065Z Starting upload of logs-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057.zip 2024-08-21T00:00:47.9591224Z Finished upload of logs-test-dynamic_aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026300057.zip 2024-08-21T00:00:47.9874457Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:00:47.9874892Z with: 2024-08-21T00:00:47.9875184Z s3-bucket: gha-artifacts 2024-08-21T00:00:47.9875614Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-21T00:00:47.9876087Z retention-days: 14 2024-08-21T00:00:47.9876424Z if-no-files-found: ignore 2024-08-21T00:00:47.9876785Z path: debug-*.zip 2024-08-21T00:00:47.9877098Z name: artifact 2024-08-21T00:00:47.9877398Z region: us-east-1 2024-08-21T00:00:47.9877697Z env: 2024-08-21T00:00:47.9877972Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:47.9878430Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:47.9879147Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:47.9879788Z ##[endgroup] 2024-08-21T00:00:48.3401777Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-21T00:00:48.3686221Z ##[group]Run # shellcheck disable=SC2156 2024-08-21T00:00:48.3686703Z # shellcheck disable=SC2156 2024-08-21T00:00:48.3687491Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-21T00:00:48.3697264Z shell: /usr/bin/bash -e {0} 2024-08-21T00:00:48.3697623Z env: 2024-08-21T00:00:48.3697902Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:48.3698356Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:48.3699080Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:48.3699710Z ##[endgroup] 2024-08-21T00:00:48.6435488Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-21T00:00:48.6436064Z with: 2024-08-21T00:00:48.6436328Z env: 2024-08-21T00:00:48.6436605Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:48.6437052Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:48.6438012Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:48.6438642Z ##[endgroup] 2024-08-21T00:00:48.6458386Z ##[group]Run set -eou pipefail 2024-08-21T00:00:48.6458796Z set -eou pipefail 2024-08-21T00:00:48.6459148Z  2024-08-21T00:00:48.6459660Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-21T00:00:48.6460289Z for _ in $(seq 1440); do 2024-08-21T00:00:48.6460765Z  # Break if no ssh session exists anymore 2024-08-21T00:00:48.6461265Z  if [ "$(who)" = "" ]; then 2024-08-21T00:00:48.6461671Z  break 2024-08-21T00:00:48.6462013Z  fi 2024-08-21T00:00:48.6462309Z  echo "." 2024-08-21T00:00:48.6462830Z  sleep 5 2024-08-21T00:00:48.6463146Z done 2024-08-21T00:00:48.6472159Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:48.6472653Z env: 2024-08-21T00:00:48.6472928Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:48.6473387Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:48.6474165Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:48.6474801Z ##[endgroup] 2024-08-21T00:00:48.6502897Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-21T00:00:48.6577556Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:00:48.6578654Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:00:48.6579546Z # shellcheck disable=SC2046 2024-08-21T00:00:48.6580175Z docker stop $(docker ps -q) || true 2024-08-21T00:00:48.6580889Z # Prune all of the docker images 2024-08-21T00:00:48.6581480Z docker system prune -af 2024-08-21T00:00:48.6589942Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:00:48.6590447Z env: 2024-08-21T00:00:48.6590728Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:00:48.6591182Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:00:48.6591908Z DOCKER_CONTAINER_ID: b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:48.6592547Z ##[endgroup] 2024-08-21T00:00:49.3412055Z b5ad0927b399 2024-08-21T00:00:53.3810771Z Deleted Containers: 2024-08-21T00:00:53.3811494Z b5ad0927b3997973ba16b6bac301b7c50061297689a7c075a8e831cd18cd966d 2024-08-21T00:00:53.3812045Z 2024-08-21T00:01:01.0975915Z Deleted Images: 2024-08-21T00:01:01.0978455Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T00:01:01.0980941Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-21T00:01:01.0982475Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-21T00:01:01.0983323Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-21T00:01:01.0984175Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-21T00:01:01.0985003Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-21T00:01:01.0985844Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-21T00:01:01.0986641Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-21T00:01:01.0987609Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-21T00:01:01.0988769Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-21T00:01:01.0989598Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-21T00:01:01.0990441Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-21T00:01:01.0991264Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-21T00:01:01.0992071Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-21T00:01:01.0992878Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-21T00:01:01.0993821Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-21T00:01:01.0994893Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-21T00:01:01.0995732Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-21T00:01:01.0996561Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-21T00:01:01.0997378Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-21T00:01:01.0998430Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-21T00:01:01.0999241Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-21T00:01:01.1000056Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-21T00:01:01.1000888Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-21T00:01:01.1001705Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-21T00:01:01.1002518Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-21T00:01:01.1003327Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-21T00:01:01.1004159Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-21T00:01:01.1004990Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-21T00:01:01.1005826Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-21T00:01:01.1006643Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-21T00:01:01.1007453Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-21T00:01:01.1008269Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-21T00:01:01.1009077Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-21T00:01:01.1009889Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-21T00:01:01.1010693Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-21T00:01:01.1011515Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-21T00:01:01.1012331Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-21T00:01:01.1013147Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-21T00:01:01.1013958Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-21T00:01:01.1014777Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-21T00:01:01.1015594Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-21T00:01:01.1016414Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-21T00:01:01.1017272Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-21T00:01:01.1018140Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-21T00:01:01.1019620Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-21T00:01:01.1020732Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-21T00:01:01.1021594Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-21T00:01:01.1022414Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-21T00:01:01.1023217Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-21T00:01:01.1024024Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-21T00:01:01.1024856Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-21T00:01:01.1025677Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-21T00:01:01.1026474Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-21T00:01:01.1027380Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-21T00:01:01.1028211Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-21T00:01:01.1029038Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-21T00:01:01.1029951Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-21T00:01:01.1030778Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-21T00:01:01.1031587Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-21T00:01:01.1032403Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-21T00:01:01.1033210Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-21T00:01:01.1034041Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-21T00:01:01.1034877Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-21T00:01:01.1035716Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-21T00:01:01.1036521Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-21T00:01:01.1037785Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-21T00:01:01.1038621Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-21T00:01:01.1039443Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-21T00:01:01.1040271Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-21T00:01:01.1041100Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-21T00:01:01.1041926Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-21T00:01:01.1042724Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-21T00:01:01.1043537Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-21T00:01:01.1044354Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-21T00:01:01.1045175Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-21T00:01:01.1046010Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-21T00:01:01.1046841Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-21T00:01:01.1047693Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-21T00:01:01.1048553Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-21T00:01:01.1049368Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-21T00:01:01.1049852Z 2024-08-21T00:01:01.1050013Z Total reclaimed space: 49.95GB 2024-08-21T00:01:01.1115633Z Post job cleanup. 2024-08-21T00:01:01.1165987Z Post job cleanup. 2024-08-21T00:01:01.2054369Z [command]/usr/bin/git version 2024-08-21T00:01:01.2127256Z git version 2.40.1 2024-08-21T00:01:01.2158769Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/3c673424-fb33-4f9f-8e60-d47f96c2a9ec/.gitconfig' 2024-08-21T00:01:01.2169800Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/3c673424-fb33-4f9f-8e60-d47f96c2a9ec' before making global git config changes 2024-08-21T00:01:01.2171206Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T00:01:01.2174115Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-21T00:01:01.2230630Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T00:01:01.2277206Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T00:01:01.2657964Z Entering 'android/libs/fbjni' 2024-08-21T00:01:01.2725785Z Entering 'third_party/FP16' 2024-08-21T00:01:01.2795413Z Entering 'third_party/FXdiv' 2024-08-21T00:01:01.2864281Z Entering 'third_party/NNPACK' 2024-08-21T00:01:01.2933186Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:01:01.3002666Z Entering 'third_party/XNNPACK' 2024-08-21T00:01:01.3088328Z Entering 'third_party/benchmark' 2024-08-21T00:01:01.3159800Z Entering 'third_party/cpp-httplib' 2024-08-21T00:01:01.3228239Z Entering 'third_party/cpuinfo' 2024-08-21T00:01:01.3298215Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:01:01.3371588Z Entering 'third_party/cutlass' 2024-08-21T00:01:01.3446757Z Entering 'third_party/eigen' 2024-08-21T00:01:01.3543092Z Entering 'third_party/fbgemm' 2024-08-21T00:01:01.3585587Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:01:01.3652424Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:01:01.3720395Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:01:01.3795334Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:01:01.3862082Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:01:01.3931675Z Entering 'third_party/flatbuffers' 2024-08-21T00:01:01.4005742Z Entering 'third_party/fmt' 2024-08-21T00:01:01.4074681Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:01:01.4143439Z Entering 'third_party/gloo' 2024-08-21T00:01:01.4212058Z Entering 'third_party/googletest' 2024-08-21T00:01:01.4283598Z Entering 'third_party/ideep' 2024-08-21T00:01:01.4350435Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:01:01.4427261Z Entering 'third_party/ittapi' 2024-08-21T00:01:01.4496885Z Entering 'third_party/kineto' 2024-08-21T00:01:01.4569708Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:01:01.4638623Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:01:01.4708994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:01:01.4776941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:01:01.4845515Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:01:01.4911775Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:01:01.4984443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:01:01.5053506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:01:01.5121479Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:01:01.5192186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:01:01.5263785Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:01:01.5333191Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:01:01.5404684Z Entering 'third_party/mimalloc' 2024-08-21T00:01:01.5474537Z Entering 'third_party/nccl/nccl' 2024-08-21T00:01:01.5544434Z Entering 'third_party/nlohmann' 2024-08-21T00:01:01.5612894Z Entering 'third_party/onnx' 2024-08-21T00:01:01.5695706Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:01:01.5762691Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:01:01.5835264Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:01:01.5904699Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:01:01.5971198Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:01:01.6040899Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:01:01.6106729Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:01:01.6180169Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:01:01.6249496Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:01:01.6315422Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:01:01.6380360Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:01:01.6450037Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:01:01.6520297Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:01:01.6608318Z Entering 'third_party/pocketfft' 2024-08-21T00:01:01.6678321Z Entering 'third_party/protobuf' 2024-08-21T00:01:01.6749963Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:01:01.6816173Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:01:01.6886664Z Entering 'third_party/psimd' 2024-08-21T00:01:01.6956349Z Entering 'third_party/pthreadpool' 2024-08-21T00:01:01.7025674Z Entering 'third_party/pybind11' 2024-08-21T00:01:01.7096075Z Entering 'third_party/python-peachpy' 2024-08-21T00:01:01.7170248Z Entering 'third_party/sleef' 2024-08-21T00:01:01.7240388Z Entering 'third_party/tensorpipe' 2024-08-21T00:01:01.7308746Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:01:01.7379277Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:01:01.7446538Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:01:01.7512821Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:01:01.7576869Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:01:01.7672662Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T00:01:01.7712255Z http.https://github.com/.extraheader 2024-08-21T00:01:01.7722061Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-21T00:01:01.7769138Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T00:01:01.8139048Z Entering 'android/libs/fbjni' 2024-08-21T00:01:01.8181812Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8227083Z Entering 'third_party/FP16' 2024-08-21T00:01:01.8272179Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8316569Z Entering 'third_party/FXdiv' 2024-08-21T00:01:01.8362395Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8406834Z Entering 'third_party/NNPACK' 2024-08-21T00:01:01.8451049Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8495343Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:01:01.8539869Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8583729Z Entering 'third_party/XNNPACK' 2024-08-21T00:01:01.8627533Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8688630Z Entering 'third_party/benchmark' 2024-08-21T00:01:01.8731393Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8775326Z Entering 'third_party/cpp-httplib' 2024-08-21T00:01:01.8818932Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8863651Z Entering 'third_party/cpuinfo' 2024-08-21T00:01:01.8907700Z http.https://github.com/.extraheader 2024-08-21T00:01:01.8952648Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:01:01.8995943Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9039601Z Entering 'third_party/cutlass' 2024-08-21T00:01:01.9083688Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9134639Z Entering 'third_party/eigen' 2024-08-21T00:01:01.9178988Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9225972Z Entering 'third_party/fbgemm' 2024-08-21T00:01:01.9270394Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9313933Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:01:01.9359700Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9402723Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:01:01.9446814Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9489388Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:01:01.9532295Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9581611Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:01:01.9626689Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9670557Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:01:01.9715808Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9766354Z Entering 'third_party/flatbuffers' 2024-08-21T00:01:01.9810319Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9858951Z Entering 'third_party/fmt' 2024-08-21T00:01:01.9902654Z http.https://github.com/.extraheader 2024-08-21T00:01:01.9946748Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:01:01.9995204Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0039400Z Entering 'third_party/gloo' 2024-08-21T00:01:02.0083963Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0127678Z Entering 'third_party/googletest' 2024-08-21T00:01:02.0172486Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0217910Z Entering 'third_party/ideep' 2024-08-21T00:01:02.0263395Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0305881Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:01:02.0348424Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0400469Z Entering 'third_party/ittapi' 2024-08-21T00:01:02.0445311Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0488631Z Entering 'third_party/kineto' 2024-08-21T00:01:02.0532623Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0578224Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:01:02.0621165Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0663939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:01:02.0707258Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0754111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:01:02.0797246Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0842687Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:01:02.0885659Z http.https://github.com/.extraheader 2024-08-21T00:01:02.0930852Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:01:02.0974214Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1017095Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:01:02.1061122Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1109232Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:01:02.1152130Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1196990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:01:02.1240054Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1284341Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:01:02.1326360Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1372004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:01:02.1413773Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1462837Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:01:02.1504141Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1548452Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:01:02.1595582Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1641478Z Entering 'third_party/mimalloc' 2024-08-21T00:01:02.1685873Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1729689Z Entering 'third_party/nccl/nccl' 2024-08-21T00:01:02.1773414Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1817689Z Entering 'third_party/nlohmann' 2024-08-21T00:01:02.1863459Z http.https://github.com/.extraheader 2024-08-21T00:01:02.1908059Z Entering 'third_party/onnx' 2024-08-21T00:01:02.1951536Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2009656Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:01:02.2056580Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2099567Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:01:02.2142979Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2191190Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:01:02.2235557Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2281203Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:01:02.2324697Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2367959Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:01:02.2410160Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2453275Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:01:02.2496391Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2539085Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:01:02.2586040Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2632375Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:01:02.2674866Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2719115Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:01:02.2761661Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2804972Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:01:02.2849424Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2890439Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:01:02.2933531Z http.https://github.com/.extraheader 2024-08-21T00:01:02.2980837Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:01:02.3026902Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3075086Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:01:02.3117267Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3184134Z Entering 'third_party/pocketfft' 2024-08-21T00:01:02.3229676Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3274544Z Entering 'third_party/protobuf' 2024-08-21T00:01:02.3317866Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3363837Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:01:02.3406474Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3450467Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:01:02.3494183Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3543255Z Entering 'third_party/psimd' 2024-08-21T00:01:02.3586362Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3630337Z Entering 'third_party/pthreadpool' 2024-08-21T00:01:02.3674954Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3720102Z Entering 'third_party/pybind11' 2024-08-21T00:01:02.3764168Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3808115Z Entering 'third_party/python-peachpy' 2024-08-21T00:01:02.3852014Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3895629Z Entering 'third_party/sleef' 2024-08-21T00:01:02.3940298Z http.https://github.com/.extraheader 2024-08-21T00:01:02.3983942Z Entering 'third_party/tensorpipe' 2024-08-21T00:01:02.4027664Z http.https://github.com/.extraheader 2024-08-21T00:01:02.4071033Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:01:02.4113183Z http.https://github.com/.extraheader 2024-08-21T00:01:02.4155964Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:01:02.4198713Z http.https://github.com/.extraheader 2024-08-21T00:01:02.4242409Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:01:02.4288066Z http.https://github.com/.extraheader 2024-08-21T00:01:02.4331744Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:01:02.4375252Z http.https://github.com/.extraheader 2024-08-21T00:01:02.4416981Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:01:02.4463076Z http.https://github.com/.extraheader 2024-08-21T00:01:02.4625619Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-21T00:01:02.4648016Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-21T00:01:02.4656105Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:01:02.4656824Z ##[endgroup] 2024-08-21T00:01:09.8609708Z Cleaning up orphan processes