2024-05-19T18:21:29.3719517Z Current runner version: '2.316.1' 2024-05-19T18:21:29.3725630Z Runner name: 'i-0f3ccc189396031c1' 2024-05-19T18:21:29.3726491Z Runner group name: 'Default' 2024-05-19T18:21:29.3727370Z Machine name: 'ip-10-0-1-60' 2024-05-19T18:21:29.3732256Z ##[group]GITHUB_TOKEN Permissions 2024-05-19T18:21:29.3734198Z Actions: read 2024-05-19T18:21:29.3734789Z Attestations: read 2024-05-19T18:21:29.3735321Z Checks: read 2024-05-19T18:21:29.3735825Z Contents: read 2024-05-19T18:21:29.3736404Z Deployments: read 2024-05-19T18:21:29.3736945Z Discussions: read 2024-05-19T18:21:29.3737460Z Issues: read 2024-05-19T18:21:29.3737987Z Metadata: read 2024-05-19T18:21:29.3738501Z Packages: read 2024-05-19T18:21:29.3739025Z Pages: read 2024-05-19T18:21:29.3739604Z PullRequests: read 2024-05-19T18:21:29.3740176Z RepositoryProjects: read 2024-05-19T18:21:29.3740801Z SecurityEvents: read 2024-05-19T18:21:29.3741362Z Statuses: read 2024-05-19T18:21:29.3741877Z ##[endgroup] 2024-05-19T18:21:29.3745079Z Secret source: Actions 2024-05-19T18:21:29.3745870Z Prepare workflow directory 2024-05-19T18:21:29.4671728Z Prepare all required actions 2024-05-19T18:21:29.4839645Z Getting action download info 2024-05-19T18:21:29.6493137Z Download action repository 'pytorch/test-infra@main' (SHA:5ab7a0ff6617c37f26f7d3ce525c072902906bdf) 2024-05-19T18:21:30.2193894Z Download action repository 'pytorch/pytorch@main' (SHA:853081a8e7e84ce3f09749929bb3cb998fd95b30) 2024-05-19T18:21:40.5224837Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-05-19T18:21:40.6575801Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-05-19T18:21:40.9731789Z Getting action download info 2024-05-19T18:21:41.0639317Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-05-19T18:21:41.2514899Z Getting action download info 2024-05-19T18:21:41.3453749Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-05-19T18:21:41.5585428Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/121956 (c8c7ddfde57442cc28a394c385cab656ce9b1f5e) 2024-05-19T18:21:41.5587811Z ##[group] Inputs 2024-05-19T18:21:41.5588365Z build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:21:41.5596950Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:21:41.5606618Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:21:41.5607825Z sync-tag: 2024-05-19T18:21:41.5609764Z timeout-minutes: 240 2024-05-19T18:21:41.5610164Z use-gha: 2024-05-19T18:21:41.5610482Z dashboard-tag: 2024-05-19T18:21:41.5610871Z s3-bucket: gha-artifacts 2024-05-19T18:21:41.5611268Z aws-role-to-assume: 2024-05-19T18:21:41.5611651Z ##[endgroup] 2024-05-19T18:21:41.5612608Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:21:41.6463223Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-05-19T18:21:41.6463795Z with: 2024-05-19T18:21:41.6464406Z github-secret: *** 2024-05-19T18:21:41.6465338Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:21:41.6466338Z activate-with-label: false 2024-05-19T18:21:41.6466708Z label: with-ssh 2024-05-19T18:21:41.6467031Z remove-existing-keys: true 2024-05-19T18:21:41.6467388Z env: 2024-05-19T18:21:41.6467673Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:21:41.6468026Z ##[endgroup] 2024-05-19T18:21:41.7400103Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-05-19T18:21:41.7401745Z ciflow reference detected, attempting to extract PR number 2024-05-19T18:21:43.3427125Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-05-19T18:21:43.4042047Z No SSH keys found for user pytorch-bot[bot] 2024-05-19T18:21:43.4042709Z Grabbing public ssh keys from https://github.com/nWEIdia.keys 2024-05-19T18:21:43.4635760Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-05-19T18:21:43.4650255Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-05-19T18:21:43.4673953Z Login using: ssh ec2-user@ec2-35-175-251-81.compute-1.amazonaws.com 2024-05-19T18:21:43.4674749Z All testing is done inside the container, to start an interactive session run: 2024-05-19T18:21:43.4675565Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-05-19T18:21:43.4916757Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-05-19T18:21:43.4917312Z with: 2024-05-19T18:21:43.4917602Z submodules: recursive 2024-05-19T18:21:43.4918260Z fetch-depth: 0 2024-05-19T18:21:43.4918628Z env: 2024-05-19T18:21:43.4918940Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:21:43.4919280Z ##[endgroup] 2024-05-19T18:21:43.5133783Z ##[group]Run retry () { 2024-05-19T18:21:43.5134208Z retry () { 2024-05-19T18:21:43.5134747Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-05-19T18:21:43.5135312Z } 2024-05-19T18:21:43.5135631Z echo "${GITHUB_WORKSPACE}" 2024-05-19T18:21:43.5136048Z if [ -z "${NO_SUDO}" ]; then 2024-05-19T18:21:43.5136536Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:21:43.5137001Z else 2024-05-19T18:21:43.5137349Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-05-19T18:21:43.5137775Z fi 2024-05-19T18:21:43.5138125Z mkdir "${GITHUB_WORKSPACE}" 2024-05-19T18:21:43.5149719Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:21:43.5150230Z env: 2024-05-19T18:21:43.5150529Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:21:43.5151005Z NO_SUDO: 2024-05-19T18:21:43.5151295Z ##[endgroup] 2024-05-19T18:21:43.5241613Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:21:43.5424043Z ##[group]Run malfet/checkout@silent-checkout 2024-05-19T18:21:43.5424489Z with: 2024-05-19T18:21:43.5424837Z ref: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:21:43.5425298Z fetch-depth: 0 2024-05-19T18:21:43.5425757Z submodules: recursive 2024-05-19T18:21:43.5426113Z quiet-checkout: true 2024-05-19T18:21:43.5426466Z repository: pytorch/pytorch 2024-05-19T18:21:43.5426962Z token: *** 2024-05-19T18:21:43.5427264Z ssh-strict: true 2024-05-19T18:21:43.5427602Z persist-credentials: true 2024-05-19T18:21:43.5427965Z clean: true 2024-05-19T18:21:43.5428311Z sparse-checkout-cone-mode: true 2024-05-19T18:21:43.5428713Z lfs: false 2024-05-19T18:21:43.5429022Z set-safe-directory: true 2024-05-19T18:21:43.5429367Z env: 2024-05-19T18:21:43.5429651Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:21:43.5429997Z ##[endgroup] 2024-05-19T18:21:43.6349854Z Syncing repository: pytorch/pytorch 2024-05-19T18:21:43.6351617Z ##[group]Getting Git version info 2024-05-19T18:21:43.6352412Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:21:43.6353357Z [command]/usr/bin/git version 2024-05-19T18:21:43.6353749Z git version 2.40.1 2024-05-19T18:21:43.6355092Z ##[endgroup] 2024-05-19T18:21:43.6366642Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7782c970-cfec-4768-ada0-7e713610baab' before making global git config changes 2024-05-19T18:21:43.6367985Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T18:21:43.6369173Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:21:43.6392751Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-05-19T18:21:43.6396693Z ##[group]Initializing the repository 2024-05-19T18:21:43.6400098Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:21:43.6417027Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-05-19T18:21:43.6418175Z hint: is subject to change. To configure the initial branch name to use in all 2024-05-19T18:21:43.6419102Z hint: of your new repositories, which will suppress this warning, call: 2024-05-19T18:21:43.6419780Z hint: 2024-05-19T18:21:43.6420312Z hint: git config --global init.defaultBranch 2024-05-19T18:21:43.6420824Z hint: 2024-05-19T18:21:43.6421435Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-05-19T18:21:43.6422395Z hint: 'development'. The just-created branch can be renamed via this command: 2024-05-19T18:21:43.6423242Z hint: 2024-05-19T18:21:43.6423614Z hint: git branch -m 2024-05-19T18:21:43.6424468Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-05-19T18:21:43.6426323Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-05-19T18:21:43.6450104Z ##[endgroup] 2024-05-19T18:21:43.6450720Z ##[group]Disabling automatic garbage collection 2024-05-19T18:21:43.6453637Z [command]/usr/bin/git config --local gc.auto 0 2024-05-19T18:21:43.6474927Z ##[endgroup] 2024-05-19T18:21:43.6475453Z ##[group]Setting up auth 2024-05-19T18:21:43.6481476Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T18:21:43.6500696Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T18:21:43.6705886Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T18:21:43.6725465Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T18:21:43.6910520Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:21:43.6944010Z ##[endgroup] 2024-05-19T18:21:43.6944586Z ##[group]Fetching the repository 2024-05-19T18:21:43.6950230Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-05-19T18:21:48.1575604Z remote: Enumerating objects: 1118700 2024-05-19T18:21:48.1577038Z remote: Enumerating objects: 1119785, done. 2024-05-19T18:21:48.1578558Z remote: Counting objects: 0% (1/1085) 2024-05-19T18:21:48.1579943Z remote: Counting objects: 1% (11/1085) 2024-05-19T18:21:48.1581217Z remote: Counting objects: 2% (22/1085) 2024-05-19T18:21:48.1582259Z remote: Counting objects: 3% (33/1085) 2024-05-19T18:21:48.1583360Z remote: Counting objects: 4% (44/1085) 2024-05-19T18:21:48.1584621Z remote: Counting objects: 5% (55/1085) 2024-05-19T18:21:48.1585531Z remote: Counting objects: 6% (66/1085) 2024-05-19T18:21:48.1586254Z remote: Counting objects: 7% (76/1085) 2024-05-19T18:21:48.1586965Z remote: Counting objects: 8% (87/1085) 2024-05-19T18:21:48.1587675Z remote: Counting objects: 9% (98/1085) 2024-05-19T18:21:48.1588430Z remote: Counting objects: 10% (109/1085) 2024-05-19T18:21:48.1589079Z remote: Counting objects: 11% (120/1085) 2024-05-19T18:21:48.1589861Z remote: Counting objects: 12% (131/1085) 2024-05-19T18:21:48.1590530Z remote: Counting objects: 13% (142/1085) 2024-05-19T18:21:48.1591328Z remote: Counting objects: 14% (152/1085) 2024-05-19T18:21:48.1591990Z remote: Counting objects: 15% (163/1085) 2024-05-19T18:21:48.1592513Z remote: Counting objects: 16% (174/1085) 2024-05-19T18:21:48.1593019Z remote: Counting objects: 17% (185/1085) 2024-05-19T18:21:48.1593531Z remote: Counting objects: 18% (196/1085) 2024-05-19T18:21:48.1594102Z remote: Counting objects: 19% (207/1085) 2024-05-19T18:21:48.1594621Z remote: Counting objects: 20% (217/1085) 2024-05-19T18:21:48.1595135Z remote: Counting objects: 21% (228/1085) 2024-05-19T18:21:48.1595648Z remote: Counting objects: 22% (239/1085) 2024-05-19T18:21:48.1596172Z remote: Counting objects: 23% (250/1085) 2024-05-19T18:21:48.1596689Z remote: Counting objects: 24% (261/1085) 2024-05-19T18:21:48.1597196Z remote: Counting objects: 25% (272/1085) 2024-05-19T18:21:48.1597900Z remote: Counting objects: 26% (283/1085) 2024-05-19T18:21:48.1598419Z remote: Counting objects: 27% (293/1085) 2024-05-19T18:21:48.1599213Z remote: Counting objects: 28% (304/1085) 2024-05-19T18:21:48.1599724Z remote: Counting objects: 29% (315/1085) 2024-05-19T18:21:48.1600238Z remote: Counting objects: 30% (326/1085) 2024-05-19T18:21:48.1600759Z remote: Counting objects: 31% (337/1085) 2024-05-19T18:21:48.1601272Z remote: Counting objects: 32% (348/1085) 2024-05-19T18:21:48.1601780Z remote: Counting objects: 33% (359/1085) 2024-05-19T18:21:48.1602298Z remote: Counting objects: 34% (369/1085) 2024-05-19T18:21:48.1602814Z remote: Counting objects: 35% (380/1085) 2024-05-19T18:21:48.1603328Z remote: Counting objects: 36% (391/1085) 2024-05-19T18:21:48.1603835Z remote: Counting objects: 37% (402/1085) 2024-05-19T18:21:48.1604349Z remote: Counting objects: 38% (413/1085) 2024-05-19T18:21:48.1604869Z remote: Counting objects: 39% (424/1085) 2024-05-19T18:21:48.1605386Z remote: Counting objects: 40% (434/1085) 2024-05-19T18:21:48.1605901Z remote: Counting objects: 41% (445/1085) 2024-05-19T18:21:48.1606419Z remote: Counting objects: 42% (456/1085) 2024-05-19T18:21:48.1606932Z remote: Counting objects: 43% (467/1085) 2024-05-19T18:21:48.1607448Z remote: Counting objects: 44% (478/1085) 2024-05-19T18:21:48.1607954Z remote: Counting objects: 45% (489/1085) 2024-05-19T18:21:48.1608471Z remote: Counting objects: 46% (500/1085) 2024-05-19T18:21:48.1609098Z remote: Counting objects: 47% (510/1085) 2024-05-19T18:21:48.1609616Z remote: Counting objects: 48% (521/1085) 2024-05-19T18:21:48.1610123Z remote: Counting objects: 49% (532/1085) 2024-05-19T18:21:48.1610639Z remote: Counting objects: 50% (543/1085) 2024-05-19T18:21:48.1611153Z remote: Counting objects: 51% (554/1085) 2024-05-19T18:21:48.1611671Z remote: Counting objects: 52% (565/1085) 2024-05-19T18:21:48.1612180Z remote: Counting objects: 53% (576/1085) 2024-05-19T18:21:48.1612704Z remote: Counting objects: 54% (586/1085) 2024-05-19T18:21:48.1613221Z remote: Counting objects: 55% (597/1085) 2024-05-19T18:21:48.1613734Z remote: Counting objects: 56% (608/1085) 2024-05-19T18:21:48.1614237Z remote: Counting objects: 57% (619/1085) 2024-05-19T18:21:48.1614752Z remote: Counting objects: 58% (630/1085) 2024-05-19T18:21:48.1615271Z remote: Counting objects: 59% (641/1085) 2024-05-19T18:21:48.1615794Z remote: Counting objects: 60% (651/1085) 2024-05-19T18:21:48.1616304Z remote: Counting objects: 61% (662/1085) 2024-05-19T18:21:48.1616847Z remote: Counting objects: 62% (673/1085) 2024-05-19T18:21:48.1617369Z remote: Counting objects: 63% (684/1085) 2024-05-19T18:21:48.1617884Z remote: Counting objects: 64% (695/1085) 2024-05-19T18:21:48.1618395Z remote: Counting objects: 65% (706/1085) 2024-05-19T18:21:48.1618899Z remote: Counting objects: 66% (717/1085) 2024-05-19T18:21:48.1619422Z remote: Counting objects: 67% (727/1085) 2024-05-19T18:21:48.1619937Z remote: Counting objects: 68% (738/1085) 2024-05-19T18:21:48.1620452Z remote: Counting objects: 69% (749/1085) 2024-05-19T18:21:48.1620957Z remote: Counting objects: 70% (760/1085) 2024-05-19T18:21:48.1621470Z remote: Counting objects: 71% (771/1085) 2024-05-19T18:21:48.1621999Z remote: Counting objects: 72% (782/1085) 2024-05-19T18:21:48.1622516Z remote: Counting objects: 73% (793/1085) 2024-05-19T18:21:48.1623021Z remote: Counting objects: 74% (803/1085) 2024-05-19T18:21:48.1623545Z remote: Counting objects: 75% (814/1085) 2024-05-19T18:21:48.1624064Z remote: Counting objects: 76% (825/1085) 2024-05-19T18:21:48.1624581Z remote: Counting objects: 77% (836/1085) 2024-05-19T18:21:48.1625093Z remote: Counting objects: 78% (847/1085) 2024-05-19T18:21:48.1625678Z remote: Counting objects: 79% (858/1085) 2024-05-19T18:21:48.1626200Z remote: Counting objects: 80% (868/1085) 2024-05-19T18:21:48.1626717Z remote: Counting objects: 81% (879/1085) 2024-05-19T18:21:48.1627225Z remote: Counting objects: 82% (890/1085) 2024-05-19T18:21:48.1627739Z remote: Counting objects: 83% (901/1085) 2024-05-19T18:21:48.1628258Z remote: Counting objects: 84% (912/1085) 2024-05-19T18:21:48.1628783Z remote: Counting objects: 85% (923/1085) 2024-05-19T18:21:48.1629289Z remote: Counting objects: 86% (934/1085) 2024-05-19T18:21:48.1629804Z remote: Counting objects: 87% (944/1085) 2024-05-19T18:21:48.1630323Z remote: Counting objects: 88% (955/1085) 2024-05-19T18:21:48.1630836Z remote: Counting objects: 89% (966/1085) 2024-05-19T18:21:48.1631481Z remote: Counting objects: 90% (977/1085) 2024-05-19T18:21:48.1631999Z remote: Counting objects: 91% (988/1085) 2024-05-19T18:21:48.1632521Z remote: Counting objects: 92% (999/1085) 2024-05-19T18:21:48.1633042Z remote: Counting objects: 93% (1010/1085) 2024-05-19T18:21:48.1633579Z remote: Counting objects: 94% (1020/1085) 2024-05-19T18:21:48.1634170Z remote: Counting objects: 95% (1031/1085) 2024-05-19T18:21:48.1634714Z remote: Counting objects: 96% (1042/1085) 2024-05-19T18:21:48.1635252Z remote: Counting objects: 97% (1053/1085) 2024-05-19T18:21:48.1635869Z remote: Counting objects: 98% (1064/1085) 2024-05-19T18:21:48.1636412Z remote: Counting objects: 99% (1075/1085) 2024-05-19T18:21:48.1636945Z remote: Counting objects: 100% (1085/1085) 2024-05-19T18:21:48.1637518Z remote: Counting objects: 100% (1085/1085), done. 2024-05-19T18:21:48.1638202Z remote: Compressing objects: 0% (1/377) 2024-05-19T18:21:48.1729935Z remote: Compressing objects: 1% (4/377) 2024-05-19T18:21:48.1978723Z remote: Compressing objects: 2% (8/377) 2024-05-19T18:21:48.2170612Z remote: Compressing objects: 3% (12/377) 2024-05-19T18:21:48.2527894Z remote: Compressing objects: 4% (16/377) 2024-05-19T18:21:48.2550971Z remote: Compressing objects: 5% (19/377) 2024-05-19T18:21:48.2551577Z remote: Compressing objects: 6% (23/377) 2024-05-19T18:21:48.2552220Z remote: Compressing objects: 7% (27/377) 2024-05-19T18:21:48.2552781Z remote: Compressing objects: 8% (31/377) 2024-05-19T18:21:48.2553340Z remote: Compressing objects: 9% (34/377) 2024-05-19T18:21:48.2553902Z remote: Compressing objects: 10% (38/377) 2024-05-19T18:21:48.2604242Z remote: Compressing objects: 11% (42/377) 2024-05-19T18:21:48.2604803Z remote: Compressing objects: 12% (46/377) 2024-05-19T18:21:48.2605754Z remote: Compressing objects: 13% (50/377) 2024-05-19T18:21:48.2606413Z remote: Compressing objects: 14% (53/377) 2024-05-19T18:21:48.2606976Z remote: Compressing objects: 15% (57/377) 2024-05-19T18:21:48.2607547Z remote: Compressing objects: 16% (61/377) 2024-05-19T18:21:48.2608097Z remote: Compressing objects: 17% (65/377) 2024-05-19T18:21:48.2629234Z remote: Compressing objects: 18% (68/377) 2024-05-19T18:21:48.2635552Z remote: Compressing objects: 19% (72/377) 2024-05-19T18:21:48.2639262Z remote: Compressing objects: 20% (76/377) 2024-05-19T18:21:48.2641549Z remote: Compressing objects: 21% (80/377) 2024-05-19T18:21:48.2647129Z remote: Compressing objects: 22% (83/377) 2024-05-19T18:21:48.2652622Z remote: Compressing objects: 23% (87/377) 2024-05-19T18:21:48.2658396Z remote: Compressing objects: 24% (91/377) 2024-05-19T18:21:48.2658952Z remote: Compressing objects: 25% (95/377) 2024-05-19T18:21:48.2661065Z remote: Compressing objects: 26% (99/377) 2024-05-19T18:21:48.2663107Z remote: Compressing objects: 27% (102/377) 2024-05-19T18:21:48.2663680Z remote: Compressing objects: 28% (106/377) 2024-05-19T18:21:48.2668110Z remote: Compressing objects: 29% (110/377) 2024-05-19T18:21:48.2670571Z remote: Compressing objects: 30% (114/377) 2024-05-19T18:21:48.2673525Z remote: Compressing objects: 31% (117/377) 2024-05-19T18:21:48.2677866Z remote: Compressing objects: 32% (121/377) 2024-05-19T18:21:48.2682658Z remote: Compressing objects: 33% (125/377) 2024-05-19T18:21:48.2684310Z remote: Compressing objects: 34% (129/377) 2024-05-19T18:21:48.2687885Z remote: Compressing objects: 35% (132/377) 2024-05-19T18:21:48.2690665Z remote: Compressing objects: 36% (136/377) 2024-05-19T18:21:48.2695517Z remote: Compressing objects: 37% (140/377) 2024-05-19T18:21:48.2696992Z remote: Compressing objects: 38% (144/377) 2024-05-19T18:21:48.2700007Z remote: Compressing objects: 39% (148/377) 2024-05-19T18:21:48.2702523Z remote: Compressing objects: 40% (151/377) 2024-05-19T18:21:48.2705521Z remote: Compressing objects: 41% (155/377) 2024-05-19T18:21:48.2707744Z remote: Compressing objects: 42% (159/377) 2024-05-19T18:21:48.2709398Z remote: Compressing objects: 43% (163/377) 2024-05-19T18:21:48.2715059Z remote: Compressing objects: 44% (166/377) 2024-05-19T18:21:48.2715618Z remote: Compressing objects: 45% (170/377) 2024-05-19T18:21:48.2716169Z remote: Compressing objects: 46% (174/377) 2024-05-19T18:21:48.2716872Z remote: Compressing objects: 47% (178/377) 2024-05-19T18:21:48.2717548Z remote: Compressing objects: 48% (181/377) 2024-05-19T18:21:48.2718382Z remote: Compressing objects: 49% (185/377) 2024-05-19T18:21:48.2718924Z remote: Compressing objects: 50% (189/377) 2024-05-19T18:21:48.2719472Z remote: Compressing objects: 51% (193/377) 2024-05-19T18:21:48.2720012Z remote: Compressing objects: 52% (197/377) 2024-05-19T18:21:48.2720566Z remote: Compressing objects: 53% (200/377) 2024-05-19T18:21:48.2721109Z remote: Compressing objects: 54% (204/377) 2024-05-19T18:21:48.2721656Z remote: Compressing objects: 55% (208/377) 2024-05-19T18:21:48.2722202Z remote: Compressing objects: 56% (212/377) 2024-05-19T18:21:48.2722748Z remote: Compressing objects: 57% (215/377) 2024-05-19T18:21:48.2723289Z remote: Compressing objects: 58% (219/377) 2024-05-19T18:21:48.2723834Z remote: Compressing objects: 59% (223/377) 2024-05-19T18:21:48.2724387Z remote: Compressing objects: 60% (227/377) 2024-05-19T18:21:48.2725891Z remote: Compressing objects: 61% (230/377) 2024-05-19T18:21:48.2727294Z remote: Compressing objects: 62% (234/377) 2024-05-19T18:21:48.2727850Z remote: Compressing objects: 63% (238/377) 2024-05-19T18:21:48.2728512Z remote: Compressing objects: 64% (242/377) 2024-05-19T18:21:48.2729836Z remote: Compressing objects: 65% (246/377) 2024-05-19T18:21:48.2730387Z remote: Compressing objects: 66% (249/377) 2024-05-19T18:21:48.2731975Z remote: Compressing objects: 67% (253/377) 2024-05-19T18:21:48.2732593Z remote: Compressing objects: 68% (257/377) 2024-05-19T18:21:48.2733288Z remote: Compressing objects: 69% (261/377) 2024-05-19T18:21:48.2733867Z remote: Compressing objects: 70% (264/377) 2024-05-19T18:21:48.2734407Z remote: Compressing objects: 71% (268/377) 2024-05-19T18:21:48.2734957Z remote: Compressing objects: 72% (272/377) 2024-05-19T18:21:48.2735518Z remote: Compressing objects: 73% (276/377) 2024-05-19T18:21:48.2736491Z remote: Compressing objects: 74% (279/377) 2024-05-19T18:21:48.2737948Z remote: Compressing objects: 75% (283/377) 2024-05-19T18:21:48.2738592Z remote: Compressing objects: 76% (287/377) 2024-05-19T18:21:48.2739142Z remote: Compressing objects: 77% (291/377) 2024-05-19T18:21:48.2739687Z remote: Compressing objects: 78% (295/377) 2024-05-19T18:21:48.2740341Z remote: Compressing objects: 79% (298/377) 2024-05-19T18:21:48.2740886Z remote: Compressing objects: 80% (302/377) 2024-05-19T18:21:48.2741439Z remote: Compressing objects: 81% (306/377) 2024-05-19T18:21:48.2741985Z remote: Compressing objects: 82% (310/377) 2024-05-19T18:21:48.2743429Z remote: Compressing objects: 83% (313/377) 2024-05-19T18:21:48.2744086Z remote: Compressing objects: 84% (317/377) 2024-05-19T18:21:48.2744681Z remote: Compressing objects: 85% (321/377) 2024-05-19T18:21:48.2745249Z remote: Compressing objects: 86% (325/377) 2024-05-19T18:21:48.2745816Z remote: Compressing objects: 87% (328/377) 2024-05-19T18:21:48.2746370Z remote: Compressing objects: 88% (332/377) 2024-05-19T18:21:48.2746929Z remote: Compressing objects: 89% (336/377) 2024-05-19T18:21:48.2749953Z remote: Compressing objects: 90% (340/377) 2024-05-19T18:21:48.2750614Z remote: Compressing objects: 91% (344/377) 2024-05-19T18:21:48.2751490Z remote: Compressing objects: 92% (347/377) 2024-05-19T18:21:48.2752047Z remote: Compressing objects: 93% (351/377) 2024-05-19T18:21:48.2752590Z remote: Compressing objects: 94% (355/377) 2024-05-19T18:21:48.2753137Z remote: Compressing objects: 95% (359/377) 2024-05-19T18:21:48.2753691Z remote: Compressing objects: 96% (362/377) 2024-05-19T18:21:48.2754282Z remote: Compressing objects: 97% (366/377) 2024-05-19T18:21:48.2755136Z remote: Compressing objects: 98% (370/377) 2024-05-19T18:21:48.2755689Z remote: Compressing objects: 99% (374/377) 2024-05-19T18:21:48.2756234Z remote: Compressing objects: 100% (377/377) 2024-05-19T18:21:48.2756829Z remote: Compressing objects: 100% (377/377), done. 2024-05-19T18:22:08.5307037Z remote: Total 1119785 (delta 762), reused 961 (delta 698), pack-reused 1118700 2024-05-19T18:22:32.4845971Z [command]/usr/bin/git rev-parse --verify --quiet c8c7ddfde57442cc28a394c385cab656ce9b1f5e^{object} 2024-05-19T18:22:32.4863192Z c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:22:32.4867271Z ##[endgroup] 2024-05-19T18:22:32.4867833Z ##[group]Determining the checkout info 2024-05-19T18:22:32.4868830Z ##[endgroup] 2024-05-19T18:22:32.4870530Z ##[group]Checking out the ref 2024-05-19T18:22:32.4873278Z [command]/usr/bin/git checkout --quiet --force c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:22:33.6977973Z ##[endgroup] 2024-05-19T18:22:33.6978600Z ##[group]Setting up auth for fetching submodules 2024-05-19T18:22:33.6982227Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-05-19T18:22:33.7016807Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-05-19T18:22:33.7034869Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-05-19T18:22:33.7052809Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-05-19T18:22:33.7075050Z ##[endgroup] 2024-05-19T18:22:33.7075618Z ##[group]Fetching submodules 2024-05-19T18:22:33.7079298Z [command]/usr/bin/git submodule sync --recursive 2024-05-19T18:22:33.7284923Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-05-19T18:22:33.7485473Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-05-19T18:22:33.7487060Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-05-19T18:22:33.7488549Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-05-19T18:22:33.7489998Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-05-19T18:22:33.7491621Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2024-05-19T18:22:33.7493391Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-05-19T18:22:33.7495127Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-05-19T18:22:33.7496679Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-05-19T18:22:33.7499004Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-05-19T18:22:33.7500855Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-05-19T18:22:33.7503420Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-05-19T18:22:33.7505605Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-05-19T18:22:33.7507887Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-05-19T18:22:33.7510312Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-05-19T18:22:33.7513145Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-05-19T18:22:33.7515599Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-05-19T18:22:33.7518507Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-05-19T18:22:33.7521673Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:22:33.7524198Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-05-19T18:22:33.7527122Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-05-19T18:22:33.7529913Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-05-19T18:22:33.7533249Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2024-05-19T18:22:33.7536289Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-05-19T18:22:33.7539371Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-05-19T18:22:33.7542611Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-05-19T18:22:33.7545839Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-05-19T18:22:33.7549210Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2024-05-19T18:22:33.7552730Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-05-19T18:22:33.7556140Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-05-19T18:22:33.7560314Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2024-05-19T18:22:33.7564481Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-05-19T18:22:33.7567752Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-05-19T18:22:33.7571528Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-05-19T18:22:33.7575006Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-05-19T18:22:33.7578905Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-05-19T18:22:33.7582699Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-05-19T18:22:33.7586792Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-05-19T18:22:33.7590738Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-05-19T18:22:33.7595021Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2024-05-19T18:22:33.7599891Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-05-19T18:22:33.7619501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-05-19T18:22:34.0145482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-05-19T18:22:34.1859228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-05-19T18:22:34.3500367Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-05-19T18:22:34.5594106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2024-05-19T18:22:34.7667280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-05-19T18:22:36.6784129Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-05-19T18:22:45.5301807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-05-19T18:22:45.9519970Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-05-19T18:22:46.4438126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-05-19T18:22:47.0096218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-05-19T18:22:48.1851435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-05-19T18:23:41.8967227Z ##[error]error: RPC failed; curl 18 Transferred a partial file 2024-05-19T18:23:41.8968235Z ##[error]error: 28 bytes of body are still expected 2024-05-19T18:23:41.8969062Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:23:41.8969768Z ##[error]fatal: early EOF 2024-05-19T18:23:41.8970423Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:23:41.8972527Z ##[error]fatal: clone of 'https://github.com/NVIDIA/cutlass.git' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass' failed 2024-05-19T18:23:41.8973821Z Failed to clone 'third_party/cutlass'. Retry scheduled 2024-05-19T18:23:41.8984949Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-05-19T18:23:47.5704478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-05-19T18:23:48.8619537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-05-19T18:23:50.4191035Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-05-19T18:23:51.4931030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-05-19T18:23:51.6295677Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-05-19T18:23:51.9987974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-05-19T18:23:52.3783080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-05-19T18:23:53.2753462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-05-19T18:23:53.5829631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2024-05-19T18:23:53.7379437Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-05-19T18:23:53.9679772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-05-19T18:23:55.4082171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-05-19T18:23:56.0504796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-05-19T18:23:56.6471044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2024-05-19T18:23:56.9093284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-05-19T18:24:02.4006792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-05-19T18:24:04.7753686Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2024-05-19T18:24:05.1977220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-05-19T18:24:08.9465232Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-05-19T18:24:09.1414348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-05-19T18:24:16.6068654Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-05-19T18:24:16.7558098Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-05-19T18:24:16.9538163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-05-19T18:24:17.8879821Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-05-19T18:24:55.0684930Z ##[error]error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:24:55.0685953Z ##[error]error: 4510 bytes of body are still expected 2024-05-19T18:24:55.0702108Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:24:55.0703115Z ##[error]fatal: early EOF 2024-05-19T18:24:55.0704086Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:24:55.0715497Z ##[error]fatal: clone of 'https://github.com/malfet/PeachPy.git' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy' failed 2024-05-19T18:24:55.0716905Z Failed to clone 'third_party/python-peachpy'. Retry scheduled 2024-05-19T18:24:55.0735110Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-05-19T18:24:55.6594688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2024-05-19T18:24:57.5931529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-05-19T18:24:57.9589756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-05-19T18:24:59.5966415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-05-19T18:24:59.8506532Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-05-19T18:24:59.8585720Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-05-19T18:24:59.8644898Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-05-19T18:24:59.8823596Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-05-19T18:24:59.9002473Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2024-05-19T18:24:59.9287720Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-05-19T18:25:00.6992380Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-05-19T18:25:00.7166684Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-05-19T18:25:00.7520807Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-05-19T18:25:00.8329389Z Submodule path 'third_party/cpuinfo': checked out 'd6860c477c99f1fce9e28eb206891af3c0e1a1d7' 2024-05-19T18:25:00.8561331Z Submodule path 'third_party/cudnn_frontend': checked out '150798fe976556078f443fdb059a1ff0361f58a2' 2024-05-19T18:25:01.2790015Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-05-19T18:25:01.4894094Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-05-19T18:25:01.5475986Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-05-19T18:25:01.5487812Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:25:01.5489503Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:25:01.5491628Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:25:01.5493481Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:25:01.5495479Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:25:01.5512510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-05-19T18:25:02.4662615Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-05-19T18:25:03.0518537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-05-19T18:25:04.7271814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-05-19T18:25:05.6324916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-05-19T18:25:05.9888087Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-05-19T18:25:06.0719487Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-05-19T18:25:06.4109069Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-05-19T18:25:06.4648408Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-05-19T18:25:06.4731767Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-05-19T18:25:06.5654924Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-05-19T18:25:06.5954377Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-05-19T18:25:06.6016365Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-05-19T18:25:06.6316071Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-05-19T18:25:06.6507980Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-05-19T18:25:06.6876245Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:25:06.6972234Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-05-19T18:25:06.6982905Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-05-19T18:25:06.6998986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-05-19T18:25:17.3853824Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-05-19T18:25:17.3923202Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2024-05-19T18:25:17.4049372Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-05-19T18:25:17.4822326Z Submodule path 'third_party/kineto': checked out '3a81076cc97092666f319846f32f36b73ce2293e' 2024-05-19T18:25:17.4833017Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:25:17.4835215Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:25:17.4837313Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:25:17.4853911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-05-19T18:25:18.0081465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-05-19T18:25:19.0795814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-05-19T18:25:20.0756633Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-05-19T18:25:20.0766528Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:25:20.0769179Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:25:20.0771825Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:25:20.0774565Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:25:20.0777306Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:25:20.0780184Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:25:20.0783044Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:25:20.0785625Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:25:20.0804225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-05-19T18:25:21.1293568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-05-19T18:25:21.4769248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-05-19T18:25:22.5479626Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-05-19T18:25:22.8172728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-05-19T18:25:23.1315296Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-05-19T18:25:24.0990398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-05-19T18:25:29.5675519Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-05-19T18:25:29.9036037Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-05-19T18:25:29.9173305Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-05-19T18:25:29.9454023Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-05-19T18:25:29.9549335Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-05-19T18:25:29.9560063Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:25:29.9577670Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-05-19T18:25:30.2217348Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-05-19T18:25:30.2353455Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-05-19T18:25:30.2693760Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-05-19T18:25:30.3464052Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-05-19T18:25:30.3581778Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-05-19T18:25:30.3866320Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-05-19T18:25:30.4356106Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-05-19T18:25:30.4653375Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-05-19T18:25:30.4843825Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-05-19T18:25:30.4944500Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2024-05-19T18:25:30.5796600Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-05-19T18:25:30.8505430Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-05-19T18:25:30.8532058Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-05-19T18:25:30.8534290Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-05-19T18:25:30.8549242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-05-19T18:25:31.2249523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-05-19T18:25:32.2544756Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-05-19T18:25:32.2815808Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-05-19T18:25:32.2951089Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2024-05-19T18:25:32.2960183Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:25:32.2975430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2024-05-19T18:25:34.8484813Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2024-05-19T18:25:34.8498373Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:25:34.8500443Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:25:34.8519223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-05-19T18:26:00.7215172Z ##[error]error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:26:00.7216214Z ##[error]error: 6719 bytes of body are still expected 2024-05-19T18:26:00.7228732Z fetch-pack: unexpected disconnect while reading sideband packet 2024-05-19T18:26:00.7230965Z ##[error]fatal: early EOF 2024-05-19T18:26:00.7231762Z ##[error]fatal: fetch-pack: invalid index-pack output 2024-05-19T18:26:00.7240383Z ##[error]fatal: clone of 'https://github.com/google/benchmark.git' into submodule path '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' failed 2024-05-19T18:26:00.7241981Z Failed to clone 'third_party/benchmark'. Retry scheduled 2024-05-19T18:26:00.7256005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2024-05-19T18:26:01.6331935Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2024-05-19T18:26:01.9946194Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-05-19T18:26:02.0598391Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2024-05-19T18:26:02.0607918Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:26:02.0625538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2024-05-19T18:26:11.2215299Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:26:11.2309708Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2024-05-19T18:26:11.2833607Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-05-19T18:26:11.2846100Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:26:11.2848674Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:26:11.2851400Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:26:11.2853944Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:26:11.2856908Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:26:11.2860074Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:26:11.2863042Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:26:11.2865326Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:26:11.2883258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-05-19T18:26:11.6466863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-05-19T18:26:12.6139354Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-05-19T18:26:12.9349231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-05-19T18:26:18.4714880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-05-19T18:26:18.7299569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-05-19T18:26:18.9089697Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-05-19T18:26:19.1762003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-05-19T18:26:25.1608690Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-05-19T18:26:25.1939874Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-05-19T18:26:25.2060822Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-05-19T18:26:25.2888185Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-05-19T18:26:25.2978096Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-05-19T18:26:25.3081537Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-05-19T18:26:25.3192967Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-05-19T18:26:25.3203854Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:26:25.3206106Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:26:25.3223100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-05-19T18:26:27.0552391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-05-19T18:26:28.1913237Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-05-19T18:26:28.2275519Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-05-19T18:26:28.5846835Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-05-19T18:26:28.5926345Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-05-19T18:26:28.8071426Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-05-19T18:26:28.8085073Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:26:28.8087090Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-05-19T18:26:28.8105541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-05-19T18:26:29.2039122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-05-19T18:26:30.1365976Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-05-19T18:26:30.1989439Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-05-19T18:26:30.2047968Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-05-19T18:26:30.2135698Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-05-19T18:26:30.2403752Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-05-19T18:26:30.2621410Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-05-19T18:26:30.2944960Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-05-19T18:26:30.3866098Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2024-05-19T18:26:30.4064057Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-05-19T18:26:30.4076417Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:26:30.4079268Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:26:30.4081196Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:26:30.4083370Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:26:30.4101170Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-05-19T18:26:31.3694430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-05-19T18:26:31.5781057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-05-19T18:26:32.7706135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-05-19T18:26:33.6955450Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-05-19T18:26:33.7066686Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-05-19T18:26:33.7634864Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-05-19T18:26:33.7859285Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-05-19T18:26:33.7869395Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:26:33.7886061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-05-19T18:26:33.9572647Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-05-19T18:26:34.3642493Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-05-19T18:26:34.4073448Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-05-19T18:26:34.4096966Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-05-19T18:26:34.4299830Z Entering 'android/libs/fbjni' 2024-05-19T18:26:34.4331157Z Entering 'third_party/FP16' 2024-05-19T18:26:34.4358216Z Entering 'third_party/FXdiv' 2024-05-19T18:26:34.4384656Z Entering 'third_party/NNPACK' 2024-05-19T18:26:34.4413934Z Entering 'third_party/QNNPACK' 2024-05-19T18:26:34.4441679Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:26:34.4466765Z Entering 'third_party/XNNPACK' 2024-05-19T18:26:34.4503388Z Entering 'third_party/benchmark' 2024-05-19T18:26:34.4531472Z Entering 'third_party/cpp-httplib' 2024-05-19T18:26:34.4559833Z Entering 'third_party/cpuinfo' 2024-05-19T18:26:34.4586353Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:26:34.4613356Z Entering 'third_party/cutlass' 2024-05-19T18:26:34.4648637Z Entering 'third_party/eigen' 2024-05-19T18:26:34.4678836Z Entering 'third_party/fbgemm' 2024-05-19T18:26:34.4707439Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:26:34.4733992Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:26:34.4760030Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:26:34.4791318Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:26:34.4820228Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:26:34.4846115Z Entering 'third_party/flatbuffers' 2024-05-19T18:26:34.4907312Z Entering 'third_party/fmt' 2024-05-19T18:26:34.4907849Z Entering 'third_party/foxi' 2024-05-19T18:26:34.4931957Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:26:34.4960250Z Entering 'third_party/gloo' 2024-05-19T18:26:34.4988563Z Entering 'third_party/googletest' 2024-05-19T18:26:34.5015135Z Entering 'third_party/ideep' 2024-05-19T18:26:34.5043593Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:26:34.5076655Z Entering 'third_party/ios-cmake' 2024-05-19T18:26:34.5105466Z Entering 'third_party/ittapi' 2024-05-19T18:26:34.5131787Z Entering 'third_party/kineto' 2024-05-19T18:26:34.5160933Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:26:34.5188300Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:26:34.5215821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:26:34.5244436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:26:34.5270360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:26:34.5295725Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:26:34.5325425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:26:34.5351198Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:26:34.5377358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:26:34.5406082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:26:34.5434263Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:26:34.5462522Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:26:34.5488856Z Entering 'third_party/mimalloc' 2024-05-19T18:26:34.5515188Z Entering 'third_party/nccl/nccl' 2024-05-19T18:26:34.5544654Z Entering 'third_party/neon2sse' 2024-05-19T18:26:34.5571108Z Entering 'third_party/nlohmann' 2024-05-19T18:26:34.5596193Z Entering 'third_party/onnx' 2024-05-19T18:26:34.5637532Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:26:34.5666546Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:26:34.5694968Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:26:34.5723221Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:26:34.5753477Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:26:34.5782771Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:26:34.5807784Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:26:34.5839766Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:26:34.5868528Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:26:34.5896324Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:26:34.5923294Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:26:34.5948554Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:26:34.5976194Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:26:34.6003073Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:26:34.6030073Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:26:34.6057366Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:26:34.6084909Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:26:34.6111742Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:26:34.6154953Z Entering 'third_party/pocketfft' 2024-05-19T18:26:34.6184555Z Entering 'third_party/protobuf' 2024-05-19T18:26:34.6215491Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:26:34.6240385Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:26:34.6268311Z Entering 'third_party/psimd' 2024-05-19T18:26:34.6295033Z Entering 'third_party/pthreadpool' 2024-05-19T18:26:34.6324580Z Entering 'third_party/pybind11' 2024-05-19T18:26:34.6353511Z Entering 'third_party/python-peachpy' 2024-05-19T18:26:34.6381862Z Entering 'third_party/sleef' 2024-05-19T18:26:34.6407636Z Entering 'third_party/tbb' 2024-05-19T18:26:34.6437321Z Entering 'third_party/tensorpipe' 2024-05-19T18:26:34.6465315Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:26:34.6492583Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:26:34.6518169Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:26:34.6545636Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:26:34.6570852Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:26:34.6610635Z ##[endgroup] 2024-05-19T18:26:34.6611299Z ##[group]Persisting credentials for submodules 2024-05-19T18:26:34.6615992Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-05-19T18:26:34.6819738Z Entering 'android/libs/fbjni' 2024-05-19T18:26:34.6855428Z Entering 'third_party/FP16' 2024-05-19T18:26:34.6890560Z Entering 'third_party/FXdiv' 2024-05-19T18:26:34.6925664Z Entering 'third_party/NNPACK' 2024-05-19T18:26:34.6961948Z Entering 'third_party/QNNPACK' 2024-05-19T18:26:34.6994856Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:26:34.7032144Z Entering 'third_party/XNNPACK' 2024-05-19T18:26:34.7077236Z Entering 'third_party/benchmark' 2024-05-19T18:26:34.7112421Z Entering 'third_party/cpp-httplib' 2024-05-19T18:26:34.7145172Z Entering 'third_party/cpuinfo' 2024-05-19T18:26:34.7181786Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:26:34.7213978Z Entering 'third_party/cutlass' 2024-05-19T18:26:34.7251710Z Entering 'third_party/eigen' 2024-05-19T18:26:34.7288276Z Entering 'third_party/fbgemm' 2024-05-19T18:26:34.7323042Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:26:34.7356367Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:26:34.7390700Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:26:34.7428182Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:26:34.7462539Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:26:34.7495859Z Entering 'third_party/flatbuffers' 2024-05-19T18:26:34.7534145Z Entering 'third_party/fmt' 2024-05-19T18:26:34.7568879Z Entering 'third_party/foxi' 2024-05-19T18:26:34.7602844Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:26:34.7638277Z Entering 'third_party/gloo' 2024-05-19T18:26:34.7673495Z Entering 'third_party/googletest' 2024-05-19T18:26:34.7708658Z Entering 'third_party/ideep' 2024-05-19T18:26:34.7742467Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:26:34.7782610Z Entering 'third_party/ios-cmake' 2024-05-19T18:26:34.7818497Z Entering 'third_party/ittapi' 2024-05-19T18:26:34.7851308Z Entering 'third_party/kineto' 2024-05-19T18:26:34.7886929Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:26:34.7921408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:26:34.7955633Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:26:34.7989229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:26:34.8023143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:26:34.8056075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:26:34.8089961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:26:34.8124184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:26:34.8157389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:26:34.8192187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:26:34.8228651Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:26:34.8263497Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:26:34.8299524Z Entering 'third_party/mimalloc' 2024-05-19T18:26:34.8333714Z Entering 'third_party/nccl/nccl' 2024-05-19T18:26:34.8367620Z Entering 'third_party/neon2sse' 2024-05-19T18:26:34.8402310Z Entering 'third_party/nlohmann' 2024-05-19T18:26:34.8440992Z Entering 'third_party/onnx' 2024-05-19T18:26:34.8485005Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:26:34.8517885Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:26:34.8553718Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:26:34.8588385Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:26:34.8625910Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:26:34.8661558Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:26:34.8694946Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:26:34.8731853Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:26:34.8765615Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:26:34.8799273Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:26:34.8832738Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:26:34.8865764Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:26:34.8897324Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:26:34.8927696Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:26:34.8961674Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:26:34.8991363Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:26:34.9026142Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:26:34.9058694Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:26:34.9104713Z Entering 'third_party/pocketfft' 2024-05-19T18:26:34.9138832Z Entering 'third_party/protobuf' 2024-05-19T18:26:34.9172849Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:26:34.9207125Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:26:34.9242471Z Entering 'third_party/psimd' 2024-05-19T18:26:34.9275567Z Entering 'third_party/pthreadpool' 2024-05-19T18:26:34.9309131Z Entering 'third_party/pybind11' 2024-05-19T18:26:34.9344384Z Entering 'third_party/python-peachpy' 2024-05-19T18:26:34.9377186Z Entering 'third_party/sleef' 2024-05-19T18:26:34.9411759Z Entering 'third_party/tbb' 2024-05-19T18:26:34.9444827Z Entering 'third_party/tensorpipe' 2024-05-19T18:26:34.9477350Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:26:34.9511237Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:26:34.9544442Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:26:34.9578646Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:26:34.9612272Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:26:34.9660784Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-05-19T18:26:34.9858520Z Entering 'android/libs/fbjni' 2024-05-19T18:26:34.9888399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-05-19T18:26:34.9901679Z Entering 'third_party/FP16' 2024-05-19T18:26:34.9934353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-05-19T18:26:34.9947244Z Entering 'third_party/FXdiv' 2024-05-19T18:26:34.9979938Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-05-19T18:26:34.9992240Z Entering 'third_party/NNPACK' 2024-05-19T18:26:35.0024768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-05-19T18:26:35.0036289Z Entering 'third_party/QNNPACK' 2024-05-19T18:26:35.0067209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2024-05-19T18:26:35.0081021Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:26:35.0112509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-05-19T18:26:35.0125084Z Entering 'third_party/XNNPACK' 2024-05-19T18:26:35.0156068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-05-19T18:26:35.0179219Z Entering 'third_party/benchmark' 2024-05-19T18:26:35.0212377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:26:35.0226181Z Entering 'third_party/cpp-httplib' 2024-05-19T18:26:35.0257631Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-05-19T18:26:35.0269473Z Entering 'third_party/cpuinfo' 2024-05-19T18:26:35.0301249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:26:35.0314222Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:26:35.0345772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-05-19T18:26:35.0357056Z Entering 'third_party/cutlass' 2024-05-19T18:26:35.0388364Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:26:35.0408112Z Entering 'third_party/eigen' 2024-05-19T18:26:35.0439531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-05-19T18:26:35.0454527Z Entering 'third_party/fbgemm' 2024-05-19T18:26:35.0485480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-05-19T18:26:35.0497508Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:26:35.0527317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-05-19T18:26:35.0539853Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:26:35.0570520Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-05-19T18:26:35.0581070Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:26:35.0611375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-05-19T18:26:35.0627864Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:26:35.0659618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:26:35.0670124Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:26:35.0701932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-05-19T18:26:35.0715398Z Entering 'third_party/flatbuffers' 2024-05-19T18:26:35.0747491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-05-19T18:26:35.0761579Z Entering 'third_party/fmt' 2024-05-19T18:26:35.0792321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:26:35.0806643Z Entering 'third_party/foxi' 2024-05-19T18:26:35.0837133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-05-19T18:26:35.0849536Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:26:35.0881781Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-05-19T18:26:35.0894192Z Entering 'third_party/gloo' 2024-05-19T18:26:35.0926081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-05-19T18:26:35.0938242Z Entering 'third_party/googletest' 2024-05-19T18:26:35.0969907Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:26:35.0982890Z Entering 'third_party/ideep' 2024-05-19T18:26:35.1015897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-05-19T18:26:35.1027465Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:26:35.1058520Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-05-19T18:26:35.1076610Z Entering 'third_party/ios-cmake' 2024-05-19T18:26:35.1108487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2024-05-19T18:26:35.1121416Z Entering 'third_party/ittapi' 2024-05-19T18:26:35.1152600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-05-19T18:26:35.1166412Z Entering 'third_party/kineto' 2024-05-19T18:26:35.1196082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-05-19T18:26:35.1208684Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:26:35.1238740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-05-19T18:26:35.1250651Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:26:35.1280791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-05-19T18:26:35.1293199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:26:35.1323267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-05-19T18:26:35.1334679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:26:35.1367040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-05-19T18:26:35.1379176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:26:35.1409134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-05-19T18:26:35.1421238Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:26:35.1453355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-05-19T18:26:35.1465070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:26:35.1496427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-05-19T18:26:35.1509443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:26:35.1540590Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:26:35.1553452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:26:35.1586065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-05-19T18:26:35.1598508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:26:35.1628734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-05-19T18:26:35.1643407Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:26:35.1674859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-05-19T18:26:35.1686811Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:26:35.1716805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-05-19T18:26:35.1730390Z Entering 'third_party/mimalloc' 2024-05-19T18:26:35.1761886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-05-19T18:26:35.1774215Z Entering 'third_party/nccl/nccl' 2024-05-19T18:26:35.1805806Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-05-19T18:26:35.1819965Z Entering 'third_party/neon2sse' 2024-05-19T18:26:35.1851598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2024-05-19T18:26:35.1864062Z Entering 'third_party/nlohmann' 2024-05-19T18:26:35.1896441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-05-19T18:26:35.1909419Z Entering 'third_party/onnx' 2024-05-19T18:26:35.1942099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:26:35.1967460Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:26:35.1999603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:26:35.2012166Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:26:35.2043850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:26:35.2058657Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:26:35.2091040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2024-05-19T18:26:35.2102745Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:26:35.2134144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2024-05-19T18:26:35.2149712Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:26:35.2182134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:26:35.2194514Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:26:35.2226592Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:26:35.2238065Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:26:35.2268997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:26:35.2289206Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:26:35.2321938Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-05-19T18:26:35.2335429Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:26:35.2365868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:26:35.2378241Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:26:35.2410722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:26:35.2422356Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:26:35.2453120Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-05-19T18:26:35.2465220Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:26:35.2496571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-05-19T18:26:35.2510028Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:26:35.2542610Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-05-19T18:26:35.2554124Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:26:35.2584949Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-05-19T18:26:35.2597379Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:26:35.2629050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-05-19T18:26:35.2641940Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:26:35.2673595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-05-19T18:26:35.2686505Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:26:35.2716237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-05-19T18:26:35.2729021Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:26:35.2760249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-05-19T18:26:35.2788839Z Entering 'third_party/pocketfft' 2024-05-19T18:26:35.2819775Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-05-19T18:26:35.2833221Z Entering 'third_party/protobuf' 2024-05-19T18:26:35.2865576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-05-19T18:26:35.2881057Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:26:35.2911257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-05-19T18:26:35.2923794Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:26:35.2955132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:26:35.2968589Z Entering 'third_party/psimd' 2024-05-19T18:26:35.2998813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-05-19T18:26:35.3010911Z Entering 'third_party/pthreadpool' 2024-05-19T18:26:35.3042541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-05-19T18:26:35.3055742Z Entering 'third_party/pybind11' 2024-05-19T18:26:35.3087830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:26:35.3101490Z Entering 'third_party/python-peachpy' 2024-05-19T18:26:35.3132109Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-05-19T18:26:35.3144407Z Entering 'third_party/sleef' 2024-05-19T18:26:35.3175190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-05-19T18:26:35.3188904Z Entering 'third_party/tbb' 2024-05-19T18:26:35.3221124Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2024-05-19T18:26:35.3235302Z Entering 'third_party/tensorpipe' 2024-05-19T18:26:35.3268857Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-05-19T18:26:35.3282723Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:26:35.3312644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-05-19T18:26:35.3324625Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:26:35.3355320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-05-19T18:26:35.3368465Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:26:35.3399210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-05-19T18:26:35.3411466Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:26:35.3443064Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-05-19T18:26:35.3454727Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:26:35.3485900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-05-19T18:26:35.4248381Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-05-19T18:26:35.4453068Z Entering 'android/libs/fbjni' 2024-05-19T18:26:35.4479007Z Entering 'third_party/FP16' 2024-05-19T18:26:35.4503103Z Entering 'third_party/FXdiv' 2024-05-19T18:26:35.4528328Z Entering 'third_party/NNPACK' 2024-05-19T18:26:35.4553490Z Entering 'third_party/QNNPACK' 2024-05-19T18:26:35.4578075Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:26:35.4602666Z Entering 'third_party/XNNPACK' 2024-05-19T18:26:35.4638525Z Entering 'third_party/benchmark' 2024-05-19T18:26:35.4663011Z Entering 'third_party/cpp-httplib' 2024-05-19T18:26:35.4688179Z Entering 'third_party/cpuinfo' 2024-05-19T18:26:35.4712829Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:26:35.4737799Z Entering 'third_party/cutlass' 2024-05-19T18:26:35.4767713Z Entering 'third_party/eigen' 2024-05-19T18:26:35.4794030Z Entering 'third_party/fbgemm' 2024-05-19T18:26:35.4823237Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:26:35.4848926Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:26:35.4879240Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:26:35.4912568Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:26:35.4939235Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:26:35.4968659Z Entering 'third_party/flatbuffers' 2024-05-19T18:26:35.4999078Z Entering 'third_party/fmt' 2024-05-19T18:26:35.5024162Z Entering 'third_party/foxi' 2024-05-19T18:26:35.5051831Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:26:35.5081357Z Entering 'third_party/gloo' 2024-05-19T18:26:35.5108639Z Entering 'third_party/googletest' 2024-05-19T18:26:35.5137024Z Entering 'third_party/ideep' 2024-05-19T18:26:35.5165299Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:26:35.5197397Z Entering 'third_party/ios-cmake' 2024-05-19T18:26:35.5226399Z Entering 'third_party/ittapi' 2024-05-19T18:26:35.5255642Z Entering 'third_party/kineto' 2024-05-19T18:26:35.5282116Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:26:35.5309402Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:26:35.5338334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:26:35.5363709Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:26:35.5391283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:26:35.5416776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:26:35.5445756Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:26:35.5472765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:26:35.5499608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:26:35.5527486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:26:35.5556858Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:26:35.5582540Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:26:35.5611984Z Entering 'third_party/mimalloc' 2024-05-19T18:26:35.5641522Z Entering 'third_party/nccl/nccl' 2024-05-19T18:26:35.5670039Z Entering 'third_party/neon2sse' 2024-05-19T18:26:35.5696945Z Entering 'third_party/nlohmann' 2024-05-19T18:26:35.5726872Z Entering 'third_party/onnx' 2024-05-19T18:26:35.5765803Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:26:35.5793384Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:26:35.5823632Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:26:35.5850808Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:26:35.5882551Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:26:35.5910007Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:26:35.5935456Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:26:35.5967744Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:26:35.5996178Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:26:35.6024205Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:26:35.6051145Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:26:35.6075992Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:26:35.6105462Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:26:35.6133025Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:26:35.6157751Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:26:35.6185123Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:26:35.6212901Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:26:35.6241393Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:26:35.6282227Z Entering 'third_party/pocketfft' 2024-05-19T18:26:35.6308520Z Entering 'third_party/protobuf' 2024-05-19T18:26:35.6337877Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:26:35.6363660Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:26:35.6389661Z Entering 'third_party/psimd' 2024-05-19T18:26:35.6417355Z Entering 'third_party/pthreadpool' 2024-05-19T18:26:35.6446968Z Entering 'third_party/pybind11' 2024-05-19T18:26:35.6472478Z Entering 'third_party/python-peachpy' 2024-05-19T18:26:35.6500310Z Entering 'third_party/sleef' 2024-05-19T18:26:35.6526763Z Entering 'third_party/tbb' 2024-05-19T18:26:35.6554937Z Entering 'third_party/tensorpipe' 2024-05-19T18:26:35.6583820Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:26:35.6611461Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:26:35.6638200Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:26:35.6664980Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:26:35.6690425Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:26:35.6728095Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-05-19T18:26:35.6930485Z Entering 'android/libs/fbjni' 2024-05-19T18:26:35.6958651Z Entering 'third_party/FP16' 2024-05-19T18:26:35.6985860Z Entering 'third_party/FXdiv' 2024-05-19T18:26:35.7012778Z Entering 'third_party/NNPACK' 2024-05-19T18:26:35.7041320Z Entering 'third_party/QNNPACK' 2024-05-19T18:26:35.7068100Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T18:26:35.7095803Z Entering 'third_party/XNNPACK' 2024-05-19T18:26:35.7133896Z Entering 'third_party/benchmark' 2024-05-19T18:26:35.7162582Z Entering 'third_party/cpp-httplib' 2024-05-19T18:26:35.7188408Z Entering 'third_party/cpuinfo' 2024-05-19T18:26:35.7216152Z Entering 'third_party/cudnn_frontend' 2024-05-19T18:26:35.7241352Z Entering 'third_party/cutlass' 2024-05-19T18:26:35.7273982Z Entering 'third_party/eigen' 2024-05-19T18:26:35.7303736Z Entering 'third_party/fbgemm' 2024-05-19T18:26:35.7331278Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T18:26:35.7357367Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T18:26:35.7385620Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T18:26:35.7416150Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T18:26:35.7443363Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T18:26:35.7471643Z Entering 'third_party/flatbuffers' 2024-05-19T18:26:35.7502836Z Entering 'third_party/fmt' 2024-05-19T18:26:35.7529419Z Entering 'third_party/foxi' 2024-05-19T18:26:35.7556256Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T18:26:35.7584687Z Entering 'third_party/gloo' 2024-05-19T18:26:35.7611356Z Entering 'third_party/googletest' 2024-05-19T18:26:35.7644488Z Entering 'third_party/ideep' 2024-05-19T18:26:35.7670451Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T18:26:35.7702071Z Entering 'third_party/ios-cmake' 2024-05-19T18:26:35.7727692Z Entering 'third_party/ittapi' 2024-05-19T18:26:35.7755824Z Entering 'third_party/kineto' 2024-05-19T18:26:35.7782039Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T18:26:35.7809983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T18:26:35.7837345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T18:26:35.7864247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T18:26:35.7892303Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T18:26:35.7918144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T18:26:35.7945847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T18:26:35.7973847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T18:26:35.8002398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T18:26:35.8029334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T18:26:35.8058012Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T18:26:35.8083565Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T18:26:35.8111829Z Entering 'third_party/mimalloc' 2024-05-19T18:26:35.8140945Z Entering 'third_party/nccl/nccl' 2024-05-19T18:26:35.8166524Z Entering 'third_party/neon2sse' 2024-05-19T18:26:35.8193730Z Entering 'third_party/nlohmann' 2024-05-19T18:26:35.8224034Z Entering 'third_party/onnx' 2024-05-19T18:26:35.8261788Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T18:26:35.8290798Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T18:26:35.8318484Z Entering 'third_party/onnx-tensorrt' 2024-05-19T18:26:35.8345573Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T18:26:35.8374286Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T18:26:35.8401962Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T18:26:35.8428128Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T18:26:35.8458838Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T18:26:35.8488304Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T18:26:35.8515092Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T18:26:35.8543415Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T18:26:35.8568841Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T18:26:35.8598067Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T18:26:35.8623614Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T18:26:35.8650293Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T18:26:35.8677047Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T18:26:35.8706675Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T18:26:35.8734107Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T18:26:35.8776854Z Entering 'third_party/pocketfft' 2024-05-19T18:26:35.8805346Z Entering 'third_party/protobuf' 2024-05-19T18:26:35.8835597Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T18:26:35.8865975Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T18:26:35.8892398Z Entering 'third_party/psimd' 2024-05-19T18:26:35.8917426Z Entering 'third_party/pthreadpool' 2024-05-19T18:26:35.8945697Z Entering 'third_party/pybind11' 2024-05-19T18:26:35.8973446Z Entering 'third_party/python-peachpy' 2024-05-19T18:26:35.9002622Z Entering 'third_party/sleef' 2024-05-19T18:26:35.9029038Z Entering 'third_party/tbb' 2024-05-19T18:26:35.9057302Z Entering 'third_party/tensorpipe' 2024-05-19T18:26:35.9085023Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T18:26:35.9111578Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T18:26:35.9137005Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T18:26:35.9162709Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T18:26:35.9189146Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T18:26:35.9228313Z ##[endgroup] 2024-05-19T18:26:35.9256249Z [command]/usr/bin/git log -1 --format='%H' 2024-05-19T18:26:35.9278225Z 'c8c7ddfde57442cc28a394c385cab656ce9b1f5e' 2024-05-19T18:26:35.9449676Z Prepare all required actions 2024-05-19T18:26:35.9450132Z Getting action download info 2024-05-19T18:26:36.0985583Z ##[group]Run ./.github/actions/setup-linux 2024-05-19T18:26:36.0986027Z env: 2024-05-19T18:26:36.0986318Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:36.0986668Z ##[endgroup] 2024-05-19T18:26:36.1046834Z ##[group]Run set -euo pipefail 2024-05-19T18:26:36.1047260Z set -euo pipefail 2024-05-19T18:26:36.1047653Z function get_ec2_metadata() { 2024-05-19T18:26:36.1048180Z  # Pulled from instance metadata endpoint for EC2 2024-05-19T18:26:36.1049063Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-05-19T18:26:36.1049831Z  category=$1 2024-05-19T18:26:36.1050363Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-05-19T18:26:36.1051004Z  runner_name_str=i-0f3ccc189396031c1 2024-05-19T18:26:36.1051497Z  if [[ -f /.inarc ]]; then 2024-05-19T18:26:36.1051995Z  echo "ARC Runner, no info on ec2 metadata" 2024-05-19T18:26:36.1052561Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-05-19T18:26:36.1053260Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-05-19T18:26:36.1053869Z  else 2024-05-19T18:26:36.1054374Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-05-19T18:26:36.1054946Z  fi 2024-05-19T18:26:36.1055241Z } 2024-05-19T18:26:36.1055611Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-05-19T18:26:36.1056203Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-05-19T18:26:36.1056856Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-05-19T18:26:36.1057429Z echo "system info $(uname -a)" 2024-05-19T18:26:36.1065224Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:36.1065723Z env: 2024-05-19T18:26:36.1066010Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:36.1066367Z ##[endgroup] 2024-05-19T18:26:36.1134969Z ami-id: ami-0ce0c36d7a00b20e2 2024-05-19T18:26:36.1174551Z instance-id: i-0f3ccc189396031c1 2024-05-19T18:26:36.1213060Z instance-type: g5.4xlarge 2024-05-19T18:26:36.1219112Z system info Linux ip-10-0-1-60.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-05-19T18:26:36.1239397Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:26:36.1240351Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-05-19T18:26:36.1248244Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:36.1248743Z env: 2024-05-19T18:26:36.1249040Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:36.1249397Z ##[endgroup] 2024-05-19T18:26:36.1333237Z ##[group]Run if systemctl is-active --quiet docker; then 2024-05-19T18:26:36.1333857Z if systemctl is-active --quiet docker; then 2024-05-19T18:26:36.1334394Z  echo "Docker daemon is running..."; 2024-05-19T18:26:36.1334845Z else 2024-05-19T18:26:36.1335335Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-05-19T18:26:36.1336024Z fi 2024-05-19T18:26:36.1343517Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:36.1344020Z env: 2024-05-19T18:26:36.1344307Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:36.1344658Z ##[endgroup] 2024-05-19T18:26:36.1380966Z Docker daemon is running... 2024-05-19T18:26:36.1425360Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:26:36.1425930Z with: 2024-05-19T18:26:36.1426214Z shell: bash 2024-05-19T18:26:36.1426514Z timeout_minutes: 5 2024-05-19T18:26:36.1426843Z max_attempts: 3 2024-05-19T18:26:36.1427168Z retry_wait_seconds: 30 2024-05-19T18:26:36.1428648Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-05-19T18:26:36.1430136Z polling_interval_seconds: 1 2024-05-19T18:26:36.1430513Z warning_on_retry: true 2024-05-19T18:26:36.1431015Z continue_on_error: false 2024-05-19T18:26:36.1431355Z env: 2024-05-19T18:26:36.1431629Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:36.1431995Z AWS_RETRY_MODE: standard 2024-05-19T18:26:36.1432352Z AWS_MAX_ATTEMPTS: 5 2024-05-19T18:26:36.1432703Z AWS_DEFAULT_REGION: us-east-1 2024-05-19T18:26:36.1433066Z ##[endgroup] 2024-05-19T18:26:37.0142102Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:26:37.0142953Z Configure a credential helper to remove this warning. See 2024-05-19T18:26:37.0143791Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:26:37.0144304Z 2024-05-19T18:26:37.0144464Z Login Succeeded 2024-05-19T18:26:37.1922708Z Command completed after 1 attempt(s). 2024-05-19T18:26:37.1972813Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:26:37.1973916Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:26:37.1974911Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-05-19T18:26:37.1983588Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:37.1984163Z env: 2024-05-19T18:26:37.1984450Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.1984798Z ##[endgroup] 2024-05-19T18:26:37.2044148Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:26:37.2044916Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T18:26:37.2045511Z # shellcheck disable=SC2046 2024-05-19T18:26:37.2045989Z docker stop $(docker ps -q) || true 2024-05-19T18:26:37.2046489Z # Prune all of the docker images 2024-05-19T18:26:37.2046958Z docker system prune -af 2024-05-19T18:26:37.2053900Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:37.2054404Z env: 2024-05-19T18:26:37.2054694Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.2055043Z ##[endgroup] 2024-05-19T18:26:37.2362825Z "docker stop" requires at least 1 argument. 2024-05-19T18:26:37.2363608Z See 'docker stop --help'. 2024-05-19T18:26:37.2363874Z 2024-05-19T18:26:37.2364144Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-05-19T18:26:37.2364540Z 2024-05-19T18:26:37.2364694Z Stop one or more running containers 2024-05-19T18:26:37.2552782Z Total reclaimed space: 0B 2024-05-19T18:26:37.2584879Z ##[group]Run set +e 2024-05-19T18:26:37.2585243Z set +e 2024-05-19T18:26:37.2585560Z set -x 2024-05-19T18:26:37.2585864Z  2024-05-19T18:26:37.2586188Z PT_DOMAIN=download.pytorch.org 2024-05-19T18:26:37.2586993Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-05-19T18:26:37.2588095Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-05-19T18:26:37.2588852Z # one is returned at random 2024-05-19T18:26:37.2589520Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-05-19T18:26:37.2590033Z  2024-05-19T18:26:37.2590362Z if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:26:37.2591318Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-05-19T18:26:37.2592085Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-05-19T18:26:37.2592639Z  2024-05-19T18:26:37.2592977Z  if [ -z "${RESOLVED_IP}" ]; then 2024-05-19T18:26:37.2593539Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-05-19T18:26:37.2594054Z  exit 1 2024-05-19T18:26:37.2594369Z  fi 2024-05-19T18:26:37.2594665Z fi 2024-05-19T18:26:37.2595054Z  2024-05-19T18:26:37.2595428Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-05-19T18:26:37.2595950Z  # Clean up any old records first 2024-05-19T18:26:37.2596475Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-05-19T18:26:37.2596939Z fi 2024-05-19T18:26:37.2597217Z  2024-05-19T18:26:37.2597868Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-05-19T18:26:37.2598491Z cat /etc/hosts 2024-05-19T18:26:37.2606252Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:37.2622358Z env: 2024-05-19T18:26:37.2622676Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.2623049Z ##[endgroup] 2024-05-19T18:26:37.2643403Z + PT_DOMAIN=download.pytorch.org 2024-05-19T18:26:37.2647227Z ++ dig -4 +short download.pytorch.org 2024-05-19T18:26:37.2647671Z ++ tail -n1 2024-05-19T18:26:37.2741826Z + RESOLVED_IP=18.160.10.22 2024-05-19T18:26:37.2742402Z + '[' -z 18.160.10.22 ']' 2024-05-19T18:26:37.2743048Z + grep -r download.pytorch.org /etc/hosts 2024-05-19T18:26:37.2749060Z + echo '18.160.10.22 download.pytorch.org' 2024-05-19T18:26:37.2749696Z + sudo tee -a /etc/hosts 2024-05-19T18:26:37.2827551Z 18.160.10.22 download.pytorch.org 2024-05-19T18:26:37.2841100Z + cat /etc/hosts 2024-05-19T18:26:37.2845850Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-05-19T18:26:37.2858522Z ::1 localhost6 localhost6.localdomain6 2024-05-19T18:26:37.2859021Z 18.160.10.22 download.pytorch.org 2024-05-19T18:26:37.3009857Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-05-19T18:26:37.3010472Z with: 2024-05-19T18:26:37.3011581Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3012817Z docker-build-dir: .ci/docker 2024-05-19T18:26:37.3013204Z working-directory: . 2024-05-19T18:26:37.3013679Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.3014207Z force-push: false 2024-05-19T18:26:37.3014516Z env: 2024-05-19T18:26:37.3014803Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.3015167Z ##[endgroup] 2024-05-19T18:26:37.3032912Z ##[group]Run set -ex 2024-05-19T18:26:37.3033283Z set -ex 2024-05-19T18:26:37.3033579Z  2024-05-19T18:26:37.3034149Z # If the docker build directory or the build script doesn't exist, the action will 2024-05-19T18:26:37.3035129Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-05-19T18:26:37.3035932Z # job could then download the pre-built image as usual 2024-05-19T18:26:37.3036678Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-05-19T18:26:37.3037369Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3038281Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3038850Z  2024-05-19T18:26:37.3039363Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-05-19T18:26:37.3040110Z  exit 0 2024-05-19T18:26:37.3040418Z else 2024-05-19T18:26:37.3040796Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3041250Z fi 2024-05-19T18:26:37.3041536Z  2024-05-19T18:26:37.3042012Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-05-19T18:26:37.3042865Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-05-19T18:26:37.3043645Z  # use it as it is, but first let's extract the tag 2024-05-19T18:26:37.3044359Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-05-19T18:26:37.3045088Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3045796Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3046355Z else 2024-05-19T18:26:37.3046796Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-05-19T18:26:37.3047472Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3048369Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3049117Z fi 2024-05-19T18:26:37.3056492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:37.3056991Z env: 2024-05-19T18:26:37.3057270Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.3057622Z REPO_NAME: pytorch 2024-05-19T18:26:37.3058751Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3059956Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:26:37.3060456Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.3060969Z ##[endgroup] 2024-05-19T18:26:37.3080665Z + [[ ! -d .ci/docker ]] 2024-05-19T18:26:37.3081198Z + [[ ! -f .ci/docker/build.sh ]] 2024-05-19T18:26:37.3081709Z + echo skip=false 2024-05-19T18:26:37.3083631Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-05-19T18:26:37.3086511Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3087693Z ++ awk -F '[:,]' '{print $2}' 2024-05-19T18:26:37.3091697Z + DOCKER_TAG=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3092551Z + echo docker-tag=d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3094286Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3128170Z ##[group]Run set +e 2024-05-19T18:26:37.3128548Z set +e 2024-05-19T18:26:37.3128857Z set -x 2024-05-19T18:26:37.3129153Z  2024-05-19T18:26:37.3129443Z login() { 2024-05-19T18:26:37.3130122Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:26:37.3130846Z } 2024-05-19T18:26:37.3131122Z  2024-05-19T18:26:37.3131409Z retry () { 2024-05-19T18:26:37.3131808Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:26:37.3132266Z } 2024-05-19T18:26:37.3132541Z  2024-05-19T18:26:37.3132869Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:26:37.3133292Z  2024-05-19T18:26:37.3133763Z # Check if image already exists, if it does then skip building it 2024-05-19T18:26:37.3134477Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-05-19T18:26:37.3134989Z  exit 0 2024-05-19T18:26:37.3135382Z fi 2024-05-19T18:26:37.3135660Z  2024-05-19T18:26:37.3136174Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-05-19T18:26:37.3137011Z # be empty. The default action would be to continue rebuild the image 2024-05-19T18:26:37.3137762Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-05-19T18:26:37.3138438Z  # if we're on the base branch then use the parent commit 2024-05-19T18:26:37.3139038Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-05-19T18:26:37.3139484Z else 2024-05-19T18:26:37.3139961Z  # otherwise we're on a PR, so use the most recent base commit 2024-05-19T18:26:37.3140656Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-05-19T18:26:37.3141179Z fi 2024-05-19T18:26:37.3141466Z  2024-05-19T18:26:37.3141791Z if [[ -z "${MERGE_BASE}" ]]; then 2024-05-19T18:26:37.3142300Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3142780Z  2024-05-19T18:26:37.3143443Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-05-19T18:26:37.3144211Z  exit 0 2024-05-19T18:26:37.3144510Z fi 2024-05-19T18:26:37.3144796Z  2024-05-19T18:26:37.3145244Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-05-19T18:26:37.3146221Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-05-19T18:26:37.3147025Z  exit 1 2024-05-19T18:26:37.3147342Z fi 2024-05-19T18:26:37.3147630Z  2024-05-19T18:26:37.3148128Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-05-19T18:26:37.3149084Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-05-19T18:26:37.3149971Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-05-19T18:26:37.3151035Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-05-19T18:26:37.3152136Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-05-19T18:26:37.3152879Z fi 2024-05-19T18:26:37.3153168Z  2024-05-19T18:26:37.3153537Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-05-19T18:26:37.3160603Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:37.3161109Z env: 2024-05-19T18:26:37.3161386Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.3161761Z DOCKER_BUILD_DIR: .ci/docker 2024-05-19T18:26:37.3162240Z BASE_REVISION: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:26:37.3163532Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3164794Z DOCKER_TAG: d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.3165399Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.3165918Z ##[endgroup] 2024-05-19T18:26:37.3184575Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.3185246Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.3186135Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:26:37.3186905Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.7232196Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:26:37.7233011Z Configure a credential helper to remove this warning. See 2024-05-19T18:26:37.7233846Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:26:37.7234472Z 2024-05-19T18:26:37.7234630Z Login Succeeded 2024-05-19T18:26:37.7248008Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.9257589Z { 2024-05-19T18:26:37.9257963Z "schemaVersion": 2, 2024-05-19T18:26:37.9258691Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-05-19T18:26:37.9259488Z "config": { 2024-05-19T18:26:37.9260173Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-05-19T18:26:37.9260980Z "size": 45561, 2024-05-19T18:26:37.9261787Z "digest": "sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef" 2024-05-19T18:26:37.9262728Z }, 2024-05-19T18:26:37.9263098Z "layers": [ 2024-05-19T18:26:37.9263510Z { 2024-05-19T18:26:37.9264189Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9265056Z "size": 28584317, 2024-05-19T18:26:37.9265897Z "digest": "sha256:63e9bbe323274e77e58d77c6ab6802d247458f784222fbb07a2556d6ec74ee05" 2024-05-19T18:26:37.9266895Z }, 2024-05-19T18:26:37.9267292Z { 2024-05-19T18:26:37.9268026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9269156Z "size": 7944771, 2024-05-19T18:26:37.9269978Z "digest": "sha256:cfb3d849840ee60cee7b02bad68c1fc3c15928ebcd88f327754766b670578ed6" 2024-05-19T18:26:37.9271173Z }, 2024-05-19T18:26:37.9271537Z { 2024-05-19T18:26:37.9272185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9273095Z "size": 57593718, 2024-05-19T18:26:37.9273903Z "digest": "sha256:968831e596a6288f0fed9b8612ee4ee8e75511037c4305058805492c5162e481" 2024-05-19T18:26:37.9274840Z }, 2024-05-19T18:26:37.9275211Z { 2024-05-19T18:26:37.9275871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9276708Z "size": 187, 2024-05-19T18:26:37.9277554Z "digest": "sha256:ea310eb267ca1cab61b6b16f566cd28bfd59a741395a011f5e76716e15ba57c6" 2024-05-19T18:26:37.9278873Z }, 2024-05-19T18:26:37.9279275Z { 2024-05-19T18:26:37.9279959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9280830Z "size": 6885, 2024-05-19T18:26:37.9281660Z "digest": "sha256:3af11d09e9cd1eb9c379f0a4071231e5a5642eb728b4b33bcb76be291f3c9488" 2024-05-19T18:26:37.9282632Z }, 2024-05-19T18:26:37.9283216Z { 2024-05-19T18:26:37.9283903Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9284737Z "size": 1361380219, 2024-05-19T18:26:37.9285544Z "digest": "sha256:ebfec18059b91e56882881ac34754f917861edb5f732c395d2a1a851bbd6db46" 2024-05-19T18:26:37.9286475Z }, 2024-05-19T18:26:37.9286826Z { 2024-05-19T18:26:37.9287469Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9288357Z "size": 62686, 2024-05-19T18:26:37.9289182Z "digest": "sha256:533b4aebf16914c763b7b0de3ce657590c6f979045e9fdf1f816adaf68d8a4d3" 2024-05-19T18:26:37.9290103Z }, 2024-05-19T18:26:37.9290463Z { 2024-05-19T18:26:37.9291140Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9291962Z "size": 1685, 2024-05-19T18:26:37.9292766Z "digest": "sha256:9dd75d06a0910f28cb1e484b8808724e5a6ee570ecb8fc04631368f546b39ed9" 2024-05-19T18:26:37.9293694Z }, 2024-05-19T18:26:37.9294057Z { 2024-05-19T18:26:37.9294710Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9295574Z "size": 1523, 2024-05-19T18:26:37.9296383Z "digest": "sha256:30bfca4dd3492d60ed8035b0eeb1229897041140db117f5663465a551e25851d" 2024-05-19T18:26:37.9297323Z }, 2024-05-19T18:26:37.9297685Z { 2024-05-19T18:26:37.9298356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9299193Z "size": 2608186849, 2024-05-19T18:26:37.9300095Z "digest": "sha256:1b57ce94cad9a5097d99b8d6f7bcd51df5a162fc3c5f5686b689b76993724bc8" 2024-05-19T18:26:37.9301035Z }, 2024-05-19T18:26:37.9301403Z { 2024-05-19T18:26:37.9302049Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9303093Z "size": 86631, 2024-05-19T18:26:37.9303936Z "digest": "sha256:9ee6bdb31195dd42fd98147a75d540efe0c5708e0ecda866a0bca060084fddab" 2024-05-19T18:26:37.9304901Z }, 2024-05-19T18:26:37.9305276Z { 2024-05-19T18:26:37.9305935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9306783Z "size": 1822, 2024-05-19T18:26:37.9307605Z "digest": "sha256:8af83a173aa33e1330fdde4efc5b96334ef85736ec0725bfeedc72c227dcfcc5" 2024-05-19T18:26:37.9308559Z }, 2024-05-19T18:26:37.9308910Z { 2024-05-19T18:26:37.9309562Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9310422Z "size": 245735746, 2024-05-19T18:26:37.9311386Z "digest": "sha256:f8281676bd20d66f882b7edad2f27bd52787f63ff5c68768681df27d74d9d07c" 2024-05-19T18:26:37.9312354Z }, 2024-05-19T18:26:37.9312726Z { 2024-05-19T18:26:37.9313379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9314229Z "size": 544, 2024-05-19T18:26:37.9315041Z "digest": "sha256:748e3cfdd8d6f501456917dcc4a9e8b11092aa4e784a46b00c3a7a13274e2742" 2024-05-19T18:26:37.9315995Z }, 2024-05-19T18:26:37.9316364Z { 2024-05-19T18:26:37.9316998Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9317581Z "size": 1279, 2024-05-19T18:26:37.9318354Z "digest": "sha256:5fa8d379f04e0947dc4af19169168984b6dfe62a40efbd97ee9bc60015cb2970" 2024-05-19T18:26:37.9318965Z }, 2024-05-19T18:26:37.9319214Z { 2024-05-19T18:26:37.9319648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9320205Z "size": 484, 2024-05-19T18:26:37.9320745Z "digest": "sha256:a95b9f41fb5bbac1fd00c55419da0b54b0c1b38cd46e22b6e95293bd639615b5" 2024-05-19T18:26:37.9321367Z }, 2024-05-19T18:26:37.9321613Z { 2024-05-19T18:26:37.9322043Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9322588Z "size": 91705725, 2024-05-19T18:26:37.9323142Z "digest": "sha256:8b6d0b819d19a961789f05d3c03c73f6ee67059e9464c8bbc0a1124113a32395" 2024-05-19T18:26:37.9323744Z }, 2024-05-19T18:26:37.9323989Z { 2024-05-19T18:26:37.9324415Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9324966Z "size": 3236, 2024-05-19T18:26:37.9325630Z "digest": "sha256:c61996e733996ff4a9cf89f2a69716a6add90b5c8b1902feb91d8e0dfd0f7bc4" 2024-05-19T18:26:37.9326240Z }, 2024-05-19T18:26:37.9326492Z { 2024-05-19T18:26:37.9326921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9327471Z "size": 1905, 2024-05-19T18:26:37.9328000Z "digest": "sha256:9e69dcd7661c1bee67677240651319ec72ee977219e8097f93be00c6555c9ab5" 2024-05-19T18:26:37.9328606Z }, 2024-05-19T18:26:37.9328850Z { 2024-05-19T18:26:37.9329278Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9329852Z "size": 699, 2024-05-19T18:26:37.9330409Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:26:37.9331025Z }, 2024-05-19T18:26:37.9331265Z { 2024-05-19T18:26:37.9331698Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9332251Z "size": 2811659560, 2024-05-19T18:26:37.9332828Z "digest": "sha256:7cb10a6ae8df3c778e529febd7eef0333c2af45e2a2ed45f45967bb7b4219131" 2024-05-19T18:26:37.9333435Z }, 2024-05-19T18:26:37.9333680Z { 2024-05-19T18:26:37.9334107Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9334653Z "size": 378, 2024-05-19T18:26:37.9335181Z "digest": "sha256:07fa87a43fb8c069109d9c97743dcc016ad605a221c1df4c7d79863d0d0a2229" 2024-05-19T18:26:37.9335783Z }, 2024-05-19T18:26:37.9336027Z { 2024-05-19T18:26:37.9336449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9336988Z "size": 11887, 2024-05-19T18:26:37.9337528Z "digest": "sha256:f16e94b048018b4677a272dcb1cb347e9c90be713acffa5bdd2d63a972224ca9" 2024-05-19T18:26:37.9338235Z }, 2024-05-19T18:26:37.9338478Z { 2024-05-19T18:26:37.9338906Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9339453Z "size": 803, 2024-05-19T18:26:37.9339987Z "digest": "sha256:c857065196d6f476be67a20b1e26f3ef6e71ae427e4feee2412e7f65cc208493" 2024-05-19T18:26:37.9340586Z }, 2024-05-19T18:26:37.9340829Z { 2024-05-19T18:26:37.9341258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9341797Z "size": 106, 2024-05-19T18:26:37.9342324Z "digest": "sha256:9d5c38cad0c9a12d6b9b3070caf13ae1003594cf17c75170ccadf0f8719f93ab" 2024-05-19T18:26:37.9342931Z }, 2024-05-19T18:26:37.9343179Z { 2024-05-19T18:26:37.9343596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9344137Z "size": 503, 2024-05-19T18:26:37.9344659Z "digest": "sha256:925ac0c7ff935448f88445123dc223bb4531147eec4d0e0c34f9f05bd4e14f46" 2024-05-19T18:26:37.9345262Z }, 2024-05-19T18:26:37.9345499Z { 2024-05-19T18:26:37.9345922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9346470Z "size": 121477245, 2024-05-19T18:26:37.9347043Z "digest": "sha256:cfbf8bf4873dc1eef0b1a35dd290348be3ceb2dafccdc4820e9460ddfd8f7c54" 2024-05-19T18:26:37.9347665Z }, 2024-05-19T18:26:37.9347908Z { 2024-05-19T18:26:37.9348335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9348873Z "size": 109, 2024-05-19T18:26:37.9349400Z "digest": "sha256:01914a189c0b5a993f1fc3ddb461abe3964eba3ae55fab9eff5a53f77de19178" 2024-05-19T18:26:37.9350004Z }, 2024-05-19T18:26:37.9350249Z { 2024-05-19T18:26:37.9350671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9351312Z "size": 488, 2024-05-19T18:26:37.9351842Z "digest": "sha256:3c2451fbe84c6b4678a24659b7abe1ea3cca59da338f16b7563d6875accf6eee" 2024-05-19T18:26:37.9352448Z }, 2024-05-19T18:26:37.9352697Z { 2024-05-19T18:26:37.9353122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9353665Z "size": 298, 2024-05-19T18:26:37.9354205Z "digest": "sha256:aea8a2a73b72b6c647fafb9f4b1b3d65cb7c28172d3b7a58de3bdb6335addb00" 2024-05-19T18:26:37.9354810Z }, 2024-05-19T18:26:37.9355117Z { 2024-05-19T18:26:37.9355546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9356080Z "size": 103, 2024-05-19T18:26:37.9356620Z "digest": "sha256:c26f348bfda1e1a6e32f77a9054fa8d26cd59cf6fe8cc3ea1ff9d1816dc74d22" 2024-05-19T18:26:37.9357245Z }, 2024-05-19T18:26:37.9357493Z { 2024-05-19T18:26:37.9358098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9358645Z "size": 1471, 2024-05-19T18:26:37.9359170Z "digest": "sha256:347c41648f51ed569232ea48943ef3e1424197b90bfae5b89898f2273f64e76b" 2024-05-19T18:26:37.9359762Z }, 2024-05-19T18:26:37.9360000Z { 2024-05-19T18:26:37.9360433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9360978Z "size": 444189070, 2024-05-19T18:26:37.9361522Z "digest": "sha256:00953247955f346ee0a69eb263efa2135a9fc3e9680322e5c39cf79e4c06fab5" 2024-05-19T18:26:37.9362102Z }, 2024-05-19T18:26:37.9362349Z { 2024-05-19T18:26:37.9362784Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9363320Z "size": 162, 2024-05-19T18:26:37.9363862Z "digest": "sha256:df27b4a5ce7ca0be34e1295afceb581f6ac42c1c0888e253c481bbce25efde72" 2024-05-19T18:26:37.9364475Z }, 2024-05-19T18:26:37.9364717Z { 2024-05-19T18:26:37.9365131Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9365670Z "size": 564, 2024-05-19T18:26:37.9366204Z "digest": "sha256:acb7fcd74bf2071a5d257e338f2229b9a9747f6e96de138091ed80cfa845709f" 2024-05-19T18:26:37.9366810Z }, 2024-05-19T18:26:37.9367046Z { 2024-05-19T18:26:37.9367469Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9368095Z "size": 35867331, 2024-05-19T18:26:37.9368642Z "digest": "sha256:5bd6ac51ef5d3109aba547ec49aec04535079b866d45939fa098e33b7f20d94f" 2024-05-19T18:26:37.9369247Z }, 2024-05-19T18:26:37.9369490Z { 2024-05-19T18:26:37.9369921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9370456Z "size": 104, 2024-05-19T18:26:37.9370986Z "digest": "sha256:1795bce633b0c48054bada65aede57ae82761e922b39b354aebedd2d3fc82a01" 2024-05-19T18:26:37.9371594Z }, 2024-05-19T18:26:37.9371838Z { 2024-05-19T18:26:37.9372253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9372794Z "size": 425, 2024-05-19T18:26:37.9373313Z "digest": "sha256:e41de6016f3c0f9309d3727db3cf335f2789260369b9c02485b5a1eda2a23949" 2024-05-19T18:26:37.9373912Z }, 2024-05-19T18:26:37.9374146Z { 2024-05-19T18:26:37.9374567Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9375116Z "size": 20262012, 2024-05-19T18:26:37.9375662Z "digest": "sha256:458f5f2af8cfff41b832b1ffa0ac89536cc75afaa2b007348475812be84a3168" 2024-05-19T18:26:37.9376267Z }, 2024-05-19T18:26:37.9376508Z { 2024-05-19T18:26:37.9376931Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9377471Z "size": 437, 2024-05-19T18:26:37.9378009Z "digest": "sha256:ef99a2a9cffb05645e67c1ff7eff9d0400a37d6277a734e0534e544e6d70abc3" 2024-05-19T18:26:37.9378614Z }, 2024-05-19T18:26:37.9378855Z { 2024-05-19T18:26:37.9379273Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9379810Z "size": 699, 2024-05-19T18:26:37.9380339Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:26:37.9380948Z }, 2024-05-19T18:26:37.9381183Z { 2024-05-19T18:26:37.9381604Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9382143Z "size": 142, 2024-05-19T18:26:37.9382663Z "digest": "sha256:1ba8d08d58765b6585b8a6430120bdf33e85a118dbe004432a9515481c728a6f" 2024-05-19T18:26:37.9383256Z }, 2024-05-19T18:26:37.9383499Z { 2024-05-19T18:26:37.9383926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9384460Z "size": 136, 2024-05-19T18:26:37.9385057Z "digest": "sha256:0c67ae0247cf4129424322fdc4c7d64de5f02712a6851839df767491d40ac668" 2024-05-19T18:26:37.9385656Z }, 2024-05-19T18:26:37.9385902Z { 2024-05-19T18:26:37.9386320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9386866Z "size": 4981810470, 2024-05-19T18:26:37.9387415Z "digest": "sha256:3ebf7de5956e7374b72ff809508272a8ac6ff23b2293581bef5847cd0af1b4bb" 2024-05-19T18:26:37.9388005Z }, 2024-05-19T18:26:37.9388250Z { 2024-05-19T18:26:37.9388673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9389218Z "size": 190, 2024-05-19T18:26:37.9389745Z "digest": "sha256:a6b88127ee12fade12d3202bc71d30768c0ddc0eb46ad572bd5089b6aa1bb0e2" 2024-05-19T18:26:37.9390357Z }, 2024-05-19T18:26:37.9390610Z { 2024-05-19T18:26:37.9391119Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9391657Z "size": 565, 2024-05-19T18:26:37.9392201Z "digest": "sha256:7b0cf0b8be50e7fae7398549f658c86b1e9c348102f84ef2c230bc6af5599739" 2024-05-19T18:26:37.9392806Z }, 2024-05-19T18:26:37.9393041Z { 2024-05-19T18:26:37.9393464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9394001Z "size": 43162755, 2024-05-19T18:26:37.9394553Z "digest": "sha256:044061f29c13bd178d3113fb9ccfc367da97f96f6bf948dfcae1ff5b1e17be2f" 2024-05-19T18:26:37.9395158Z }, 2024-05-19T18:26:37.9395403Z { 2024-05-19T18:26:37.9395829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9396372Z "size": 106, 2024-05-19T18:26:37.9396891Z "digest": "sha256:f21869a483d43c856cf8639bdb4270d8295023741b5cddebed62bcfe3b8caa7b" 2024-05-19T18:26:37.9397558Z }, 2024-05-19T18:26:37.9397974Z { 2024-05-19T18:26:37.9398408Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9398944Z "size": 1212, 2024-05-19T18:26:37.9399495Z "digest": "sha256:a155d9e6e7e7fd2243fba6789a98b96aa2db9d5d6462b50d9c11cfc7f3ac34e7" 2024-05-19T18:26:37.9400109Z }, 2024-05-19T18:26:37.9400346Z { 2024-05-19T18:26:37.9400769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9401306Z "size": 699, 2024-05-19T18:26:37.9401866Z "digest": "sha256:cfcaf2eb82498ef031f7d481e635b1a37af01b9e389f30555b603d75e2c00602" 2024-05-19T18:26:37.9402468Z }, 2024-05-19T18:26:37.9402715Z { 2024-05-19T18:26:37.9403136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9403685Z "size": 137, 2024-05-19T18:26:37.9404209Z "digest": "sha256:6028f518cba8d1b51ca287446cfc0a9db237fbeea2198738ee2c114a24a8c34a" 2024-05-19T18:26:37.9404819Z }, 2024-05-19T18:26:37.9405074Z { 2024-05-19T18:26:37.9405498Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9406032Z "size": 119, 2024-05-19T18:26:37.9406559Z "digest": "sha256:9f0e16ae766e08c8bf1276b07cd6781a3f9ca54131846275ffcfc9473a6312dd" 2024-05-19T18:26:37.9407164Z }, 2024-05-19T18:26:37.9407402Z { 2024-05-19T18:26:37.9407825Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9408367Z "size": 1853517530, 2024-05-19T18:26:37.9408915Z "digest": "sha256:26d45b11322ae5593c42d7929f046ca6d236f7ec5e8a13d8930360c013d32747" 2024-05-19T18:26:37.9409499Z }, 2024-05-19T18:26:37.9409766Z { 2024-05-19T18:26:37.9410215Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9410751Z "size": 172, 2024-05-19T18:26:37.9411267Z "digest": "sha256:a78a7e4425658766ba6e70bb1367f10c352dc4a7bfb16432be3d9a3496368333" 2024-05-19T18:26:37.9411860Z }, 2024-05-19T18:26:37.9412110Z { 2024-05-19T18:26:37.9412524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9413069Z "size": 1838, 2024-05-19T18:26:37.9413617Z "digest": "sha256:df585564bf4cc28aec4957fbec751620ae4ca50edbbdcec1c5eacc306a16e2e1" 2024-05-19T18:26:37.9414232Z }, 2024-05-19T18:26:37.9414558Z { 2024-05-19T18:26:37.9414989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9415537Z "size": 7529792, 2024-05-19T18:26:37.9416073Z "digest": "sha256:f125619c47120e1a542ddb0e822b5ffc07f9d165a0977f8c4064fdf67865e3fd" 2024-05-19T18:26:37.9416662Z }, 2024-05-19T18:26:37.9416970Z + exit 0 2024-05-19T18:26:37.9417226Z { 2024-05-19T18:26:37.9417649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9418187Z "size": 162, 2024-05-19T18:26:37.9418710Z "digest": "sha256:4cc671ff8870320a4e5bab71f83b755b9b96c777dba43fdbfc1a0e3d001d8936" 2024-05-19T18:26:37.9419313Z }, 2024-05-19T18:26:37.9419571Z { 2024-05-19T18:26:37.9433598Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9434261Z "size": 7939, 2024-05-19T18:26:37.9435002Z "digest": "sha256:e1d3209e04ae7e31773f49e41f7cee2949fa017569383786876f6d1ba0d7d8ac" 2024-05-19T18:26:37.9435908Z }, 2024-05-19T18:26:37.9436180Z { 2024-05-19T18:26:37.9436616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9437160Z "size": 8072, 2024-05-19T18:26:37.9437901Z "digest": "sha256:322e57f70842c48212f87b2e4c1cc094c25470b90fec1d30ad715f4d907c1f79" 2024-05-19T18:26:37.9438508Z }, 2024-05-19T18:26:37.9438751Z { 2024-05-19T18:26:37.9439169Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9439714Z "size": 301, 2024-05-19T18:26:37.9440244Z "digest": "sha256:d58f09a95d3e18b4d653b7767e371c1e9ef320625fa4cc6a92cbd1771d7ef768" 2024-05-19T18:26:37.9440853Z }, 2024-05-19T18:26:37.9441097Z { 2024-05-19T18:26:37.9441654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9442196Z "size": 7617584, 2024-05-19T18:26:37.9442732Z "digest": "sha256:f611e28aa59ab55b92e54aac3cc04729565ccae9d0edb75cc45a534d752a9de1" 2024-05-19T18:26:37.9443344Z }, 2024-05-19T18:26:37.9443583Z { 2024-05-19T18:26:37.9444012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9444545Z "size": 108, 2024-05-19T18:26:37.9445070Z "digest": "sha256:ff67e6bd8e2302312131632eae7033ce84ce85503282bfd8b31c008ff533990a" 2024-05-19T18:26:37.9445665Z }, 2024-05-19T18:26:37.9445909Z { 2024-05-19T18:26:37.9446326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9446870Z "size": 54145477, 2024-05-19T18:26:37.9447403Z "digest": "sha256:d2b06b7f79e8770329516b40d5189f7c2b2d38404132638103a594685fe1d569" 2024-05-19T18:26:37.9447986Z }, 2024-05-19T18:26:37.9448225Z { 2024-05-19T18:26:37.9448649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9449200Z "size": 507, 2024-05-19T18:26:37.9449723Z "digest": "sha256:c27bc13ce9d41097b89fa4c7f53feb417379940ba35474fede1b3322540e85e9" 2024-05-19T18:26:37.9450327Z }, 2024-05-19T18:26:37.9450567Z { 2024-05-19T18:26:37.9450997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9451535Z "size": 1443748723, 2024-05-19T18:26:37.9452087Z "digest": "sha256:8680ed64cc192868f4e34d4dba9121684e3bba13ac365c8e8539386fe4a5147c" 2024-05-19T18:26:37.9452686Z }, 2024-05-19T18:26:37.9452928Z { 2024-05-19T18:26:37.9453350Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9453890Z "size": 106, 2024-05-19T18:26:37.9454409Z "digest": "sha256:400e9c337687d211e1a78744e24985fbaca792e988f3caf5861040d80b088662" 2024-05-19T18:26:37.9454996Z }, 2024-05-19T18:26:37.9455237Z { 2024-05-19T18:26:37.9455657Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9456210Z "size": 558, 2024-05-19T18:26:37.9456730Z "digest": "sha256:914d4657be1b6936372f45195af96a959eab5bced80ca9deeafc3d6d9db6c03c" 2024-05-19T18:26:37.9457337Z }, 2024-05-19T18:26:37.9457583Z { 2024-05-19T18:26:37.9458008Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9458644Z "size": 46248255, 2024-05-19T18:26:37.9459190Z "digest": "sha256:ecc1d0285d28b6339b9648e2195ce576f30288563ac5de0882653a42bcc0eb5b" 2024-05-19T18:26:37.9459784Z }, 2024-05-19T18:26:37.9460018Z { 2024-05-19T18:26:37.9460437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-05-19T18:26:37.9460978Z "size": 111, 2024-05-19T18:26:37.9461507Z "digest": "sha256:b882dcc190622361a09ea00b970e0fae5f81ea1f0431a45afa329fd3c97f66d9" 2024-05-19T18:26:37.9462103Z } 2024-05-19T18:26:37.9462345Z ] 2024-05-19T18:26:37.9462589Z } 2024-05-19T18:26:37.9564219Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:26:37.9564715Z tag=${ECR_DOCKER_IMAGE##*/} 2024-05-19T18:26:37.9565280Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-05-19T18:26:37.9573267Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:37.9573776Z env: 2024-05-19T18:26:37.9574066Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.9575242Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.9576424Z ##[endgroup] 2024-05-19T18:26:37.9599489Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks-d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.9648228Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-05-19T18:26:37.9648800Z with: 2024-05-19T18:26:37.9649889Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.9651299Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.9651815Z env: 2024-05-19T18:26:37.9652102Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.9652456Z ##[endgroup] 2024-05-19T18:26:37.9668401Z ##[group]Run set -x 2024-05-19T18:26:37.9668838Z set -x 2024-05-19T18:26:37.9669159Z set +e 2024-05-19T18:26:37.9669467Z  2024-05-19T18:26:37.9669754Z login() { 2024-05-19T18:26:37.9670422Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-05-19T18:26:37.9671245Z } 2024-05-19T18:26:37.9671540Z  2024-05-19T18:26:37.9671858Z retry () { 2024-05-19T18:26:37.9672261Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-05-19T18:26:37.9672716Z } 2024-05-19T18:26:37.9673003Z  2024-05-19T18:26:37.9673326Z retry login "${DOCKER_REGISTRY}" 2024-05-19T18:26:37.9673750Z  2024-05-19T18:26:37.9674038Z set -e 2024-05-19T18:26:37.9674518Z # ignore output since only exit code is used for conditional 2024-05-19T18:26:37.9675221Z # only pull docker image if it's not available locally 2024-05-19T18:26:37.9675986Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-05-19T18:26:37.9676681Z  retry docker pull "${DOCKER_IMAGE}" 2024-05-19T18:26:37.9677129Z fi 2024-05-19T18:26:37.9683967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:26:37.9684465Z env: 2024-05-19T18:26:37.9684753Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:26:37.9685921Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:37.9687220Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.9687739Z ##[endgroup] 2024-05-19T18:26:37.9711062Z + set +e 2024-05-19T18:26:37.9711685Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.9712325Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:37.9713181Z + aws ecr get-login-password --region us-east-1 2024-05-19T18:26:37.9713953Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-05-19T18:26:38.3773838Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-05-19T18:26:38.3774718Z Configure a credential helper to remove this warning. See 2024-05-19T18:26:38.3775966Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-05-19T18:26:38.3776712Z 2024-05-19T18:26:38.3776888Z Login Succeeded 2024-05-19T18:26:38.3788010Z + set -e 2024-05-19T18:26:38.3789346Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:38.3943356Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:38.3945507Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:26:38.6031275Z d55d6a6f08250cd2609d528116ab6fd6e787bee0: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks 2024-05-19T18:26:38.6032523Z 63e9bbe32327: Pulling fs layer 2024-05-19T18:26:38.6033107Z cfb3d849840e: Pulling fs layer 2024-05-19T18:26:38.6033891Z 968831e596a6: Pulling fs layer 2024-05-19T18:26:38.6034429Z ea310eb267ca: Pulling fs layer 2024-05-19T18:26:38.6034935Z 3af11d09e9cd: Pulling fs layer 2024-05-19T18:26:38.6035449Z ebfec18059b9: Pulling fs layer 2024-05-19T18:26:38.6035886Z 533b4aebf169: Pulling fs layer 2024-05-19T18:26:38.6036560Z 9dd75d06a091: Pulling fs layer 2024-05-19T18:26:38.6037078Z 30bfca4dd349: Pulling fs layer 2024-05-19T18:26:38.6037535Z 1b57ce94cad9: Pulling fs layer 2024-05-19T18:26:38.6038138Z 9ee6bdb31195: Pulling fs layer 2024-05-19T18:26:38.6038581Z 8af83a173aa3: Pulling fs layer 2024-05-19T18:26:38.6038947Z f8281676bd20: Pulling fs layer 2024-05-19T18:26:38.6039395Z 748e3cfdd8d6: Pulling fs layer 2024-05-19T18:26:38.6039780Z 5fa8d379f04e: Pulling fs layer 2024-05-19T18:26:38.6040233Z a95b9f41fb5b: Pulling fs layer 2024-05-19T18:26:38.6040745Z 8b6d0b819d19: Pulling fs layer 2024-05-19T18:26:38.6041188Z c61996e73399: Pulling fs layer 2024-05-19T18:26:38.6041590Z 9e69dcd7661c: Pulling fs layer 2024-05-19T18:26:38.6042051Z cfcaf2eb8249: Pulling fs layer 2024-05-19T18:26:38.6042433Z 7cb10a6ae8df: Pulling fs layer 2024-05-19T18:26:38.6042957Z 07fa87a43fb8: Pulling fs layer 2024-05-19T18:26:38.6043359Z f16e94b04801: Pulling fs layer 2024-05-19T18:26:38.6043830Z c857065196d6: Pulling fs layer 2024-05-19T18:26:38.6044246Z 748e3cfdd8d6: Waiting 2024-05-19T18:26:38.6044656Z 9d5c38cad0c9: Pulling fs layer 2024-05-19T18:26:38.6045020Z c61996e73399: Waiting 2024-05-19T18:26:38.6045449Z 925ac0c7ff93: Pulling fs layer 2024-05-19T18:26:38.6045821Z 7cb10a6ae8df: Waiting 2024-05-19T18:26:38.6046265Z 3af11d09e9cd: Waiting 2024-05-19T18:26:38.6046609Z cfbf8bf4873d: Pulling fs layer 2024-05-19T18:26:38.6047080Z 5fa8d379f04e: Waiting 2024-05-19T18:26:38.6047458Z 8b6d0b819d19: Waiting 2024-05-19T18:26:38.6047845Z a95b9f41fb5b: Waiting 2024-05-19T18:26:38.6048203Z 1b57ce94cad9: Waiting 2024-05-19T18:26:38.6048524Z cfcaf2eb8249: Waiting 2024-05-19T18:26:38.6048911Z 9dd75d06a091: Waiting 2024-05-19T18:26:38.6049233Z 30bfca4dd349: Waiting 2024-05-19T18:26:38.6049551Z 9ee6bdb31195: Waiting 2024-05-19T18:26:38.6049869Z 07fa87a43fb8: Waiting 2024-05-19T18:26:38.6050249Z 01914a189c0b: Pulling fs layer 2024-05-19T18:26:38.6050616Z 3c2451fbe84c: Pulling fs layer 2024-05-19T18:26:38.6050994Z aea8a2a73b72: Pulling fs layer 2024-05-19T18:26:38.6051437Z c26f348bfda1: Pulling fs layer 2024-05-19T18:26:38.6051814Z ebfec18059b9: Waiting 2024-05-19T18:26:38.6052206Z 347c41648f51: Pulling fs layer 2024-05-19T18:26:38.6052700Z f16e94b04801: Waiting 2024-05-19T18:26:38.6053110Z 00953247955f: Pulling fs layer 2024-05-19T18:26:38.6053566Z 925ac0c7ff93: Waiting 2024-05-19T18:26:38.6053896Z c26f348bfda1: Waiting 2024-05-19T18:26:38.6054272Z 8af83a173aa3: Waiting 2024-05-19T18:26:38.6054619Z df27b4a5ce7c: Pulling fs layer 2024-05-19T18:26:38.6055062Z f8281676bd20: Waiting 2024-05-19T18:26:38.6055400Z acb7fcd74bf2: Pulling fs layer 2024-05-19T18:26:38.6055851Z 5bd6ac51ef5d: Pulling fs layer 2024-05-19T18:26:38.6056232Z 1795bce633b0: Pulling fs layer 2024-05-19T18:26:38.6056677Z e41de6016f3c: Pulling fs layer 2024-05-19T18:26:38.6057038Z aea8a2a73b72: Waiting 2024-05-19T18:26:38.6057387Z cfbf8bf4873d: Waiting 2024-05-19T18:26:38.6057752Z 3c2451fbe84c: Waiting 2024-05-19T18:26:38.6058071Z 533b4aebf169: Waiting 2024-05-19T18:26:38.6058463Z 458f5f2af8cf: Pulling fs layer 2024-05-19T18:26:38.6058839Z df27b4a5ce7c: Waiting 2024-05-19T18:26:38.6059245Z ef99a2a9cffb: Pulling fs layer 2024-05-19T18:26:38.6059607Z 00953247955f: Waiting 2024-05-19T18:26:38.6059945Z 347c41648f51: Waiting 2024-05-19T18:26:38.6060305Z acb7fcd74bf2: Waiting 2024-05-19T18:26:38.6060626Z e41de6016f3c: Waiting 2024-05-19T18:26:38.6061011Z 1ba8d08d5876: Pulling fs layer 2024-05-19T18:26:38.6061379Z 01914a189c0b: Waiting 2024-05-19T18:26:38.6061759Z 458f5f2af8cf: Waiting 2024-05-19T18:26:38.6062100Z 0c67ae0247cf: Pulling fs layer 2024-05-19T18:26:38.6062512Z 3ebf7de5956e: Pulling fs layer 2024-05-19T18:26:38.6062926Z a6b88127ee12: Pulling fs layer 2024-05-19T18:26:38.6063329Z 7b0cf0b8be50: Pulling fs layer 2024-05-19T18:26:38.6063872Z 0c67ae0247cf: Waiting 2024-05-19T18:26:38.6064240Z 1ba8d08d5876: Waiting 2024-05-19T18:26:38.6064597Z 3ebf7de5956e: Waiting 2024-05-19T18:26:38.6064932Z 044061f29c13: Pulling fs layer 2024-05-19T18:26:38.6065363Z f21869a483d4: Pulling fs layer 2024-05-19T18:26:38.6065732Z a6b88127ee12: Waiting 2024-05-19T18:26:38.6066204Z 044061f29c13: Waiting 2024-05-19T18:26:38.6066530Z a155d9e6e7e7: Pulling fs layer 2024-05-19T18:26:38.6066966Z 6028f518cba8: Pulling fs layer 2024-05-19T18:26:38.6067345Z 9f0e16ae766e: Pulling fs layer 2024-05-19T18:26:38.6067777Z 26d45b11322a: Pulling fs layer 2024-05-19T18:26:38.6068148Z a78a7e442565: Pulling fs layer 2024-05-19T18:26:38.6068579Z df585564bf4c: Pulling fs layer 2024-05-19T18:26:38.6068956Z a78a7e442565: Waiting 2024-05-19T18:26:38.6069311Z f125619c4712: Pulling fs layer 2024-05-19T18:26:38.6069711Z df585564bf4c: Waiting 2024-05-19T18:26:38.6070049Z 4cc671ff8870: Pulling fs layer 2024-05-19T18:26:38.6070488Z e1d3209e04ae: Pulling fs layer 2024-05-19T18:26:38.6070966Z f125619c4712: Waiting 2024-05-19T18:26:38.6071423Z 322e57f70842: Pulling fs layer 2024-05-19T18:26:38.6071785Z 4cc671ff8870: Waiting 2024-05-19T18:26:38.6072188Z d58f09a95d3e: Pulling fs layer 2024-05-19T18:26:38.6072542Z e1d3209e04ae: Waiting 2024-05-19T18:26:38.6072937Z f611e28aa59a: Pulling fs layer 2024-05-19T18:26:38.6073330Z ff67e6bd8e23: Pulling fs layer 2024-05-19T18:26:38.6073742Z d58f09a95d3e: Waiting 2024-05-19T18:26:38.6074067Z f611e28aa59a: Waiting 2024-05-19T18:26:38.6074485Z d2b06b7f79e8: Pulling fs layer 2024-05-19T18:26:38.6074935Z c27bc13ce9d4: Pulling fs layer 2024-05-19T18:26:38.6075297Z d2b06b7f79e8: Waiting 2024-05-19T18:26:38.6075746Z 8680ed64cc19: Pulling fs layer 2024-05-19T18:26:38.6076119Z 400e9c337687: Pulling fs layer 2024-05-19T18:26:38.6076716Z c27bc13ce9d4: Waiting 2024-05-19T18:26:38.6077060Z 914d4657be1b: Pulling fs layer 2024-05-19T18:26:38.6077504Z ecc1d0285d28: Pulling fs layer 2024-05-19T18:26:38.6078057Z b882dcc19062: Pulling fs layer 2024-05-19T18:26:38.6078506Z 8680ed64cc19: Waiting 2024-05-19T18:26:38.6078943Z b882dcc19062: Waiting 2024-05-19T18:26:38.6079338Z ecc1d0285d28: Waiting 2024-05-19T18:26:38.6079777Z ff67e6bd8e23: Waiting 2024-05-19T18:26:38.7784298Z cfb3d849840e: Verifying Checksum 2024-05-19T18:26:38.7784901Z cfb3d849840e: Download complete 2024-05-19T18:26:38.8947070Z ea310eb267ca: Verifying Checksum 2024-05-19T18:26:38.8947671Z ea310eb267ca: Download complete 2024-05-19T18:26:38.9544977Z 63e9bbe32327: Verifying Checksum 2024-05-19T18:26:38.9545442Z 63e9bbe32327: Download complete 2024-05-19T18:26:38.9900920Z 3af11d09e9cd: Download complete 2024-05-19T18:26:39.0776140Z 533b4aebf169: Download complete 2024-05-19T18:26:39.1579088Z 9dd75d06a091: Verifying Checksum 2024-05-19T18:26:39.1579665Z 9dd75d06a091: Download complete 2024-05-19T18:26:39.2364767Z 968831e596a6: Verifying Checksum 2024-05-19T18:26:39.2365387Z 968831e596a6: Download complete 2024-05-19T18:26:39.2808804Z 30bfca4dd349: Verifying Checksum 2024-05-19T18:26:39.2809305Z 30bfca4dd349: Download complete 2024-05-19T18:26:39.3794639Z 9ee6bdb31195: Verifying Checksum 2024-05-19T18:26:39.3795229Z 9ee6bdb31195: Download complete 2024-05-19T18:26:39.4639362Z 8af83a173aa3: Verifying Checksum 2024-05-19T18:26:39.4639850Z 8af83a173aa3: Download complete 2024-05-19T18:26:39.6408563Z 63e9bbe32327: Pull complete 2024-05-19T18:26:39.8969340Z cfb3d849840e: Pull complete 2024-05-19T18:26:40.8246377Z 968831e596a6: Pull complete 2024-05-19T18:26:40.9312334Z ea310eb267ca: Pull complete 2024-05-19T18:26:41.0352241Z 3af11d09e9cd: Pull complete 2024-05-19T18:26:41.9811799Z f8281676bd20: Verifying Checksum 2024-05-19T18:26:41.9812379Z f8281676bd20: Download complete 2024-05-19T18:26:42.1012230Z 748e3cfdd8d6: Verifying Checksum 2024-05-19T18:26:42.1012703Z 748e3cfdd8d6: Download complete 2024-05-19T18:26:42.1836856Z 5fa8d379f04e: Verifying Checksum 2024-05-19T18:26:42.1837959Z 5fa8d379f04e: Download complete 2024-05-19T18:26:42.2664499Z a95b9f41fb5b: Verifying Checksum 2024-05-19T18:26:43.2270213Z 8b6d0b819d19: Verifying Checksum 2024-05-19T18:26:43.2270661Z 8b6d0b819d19: Download complete 2024-05-19T18:26:43.2953555Z c61996e73399: Download complete 2024-05-19T18:26:43.3767546Z 9e69dcd7661c: Download complete 2024-05-19T18:26:43.4506520Z cfcaf2eb8249: Download complete 2024-05-19T18:26:52.6304610Z ebfec18059b9: Verifying Checksum 2024-05-19T18:26:52.6305394Z ebfec18059b9: Download complete 2024-05-19T18:26:52.7123548Z 07fa87a43fb8: Verifying Checksum 2024-05-19T18:26:52.7124164Z 07fa87a43fb8: Download complete 2024-05-19T18:26:52.7914811Z f16e94b04801: Verifying Checksum 2024-05-19T18:26:52.7915248Z f16e94b04801: Download complete 2024-05-19T18:26:52.8776088Z c857065196d6: Verifying Checksum 2024-05-19T18:26:52.8776636Z c857065196d6: Download complete 2024-05-19T18:26:52.9601799Z 9d5c38cad0c9: Verifying Checksum 2024-05-19T18:26:52.9602380Z 9d5c38cad0c9: Download complete 2024-05-19T18:26:53.0446486Z 925ac0c7ff93: Verifying Checksum 2024-05-19T18:26:53.0447054Z 925ac0c7ff93: Download complete 2024-05-19T18:26:54.3237239Z cfbf8bf4873d: Verifying Checksum 2024-05-19T18:26:54.3237853Z cfbf8bf4873d: Download complete 2024-05-19T18:26:54.4042457Z 01914a189c0b: Verifying Checksum 2024-05-19T18:26:54.4042885Z 01914a189c0b: Download complete 2024-05-19T18:26:54.5164116Z 3c2451fbe84c: Download complete 2024-05-19T18:26:54.5833555Z aea8a2a73b72: Verifying Checksum 2024-05-19T18:26:54.5834089Z aea8a2a73b72: Download complete 2024-05-19T18:26:54.6749037Z c26f348bfda1: Verifying Checksum 2024-05-19T18:26:54.6749473Z c26f348bfda1: Download complete 2024-05-19T18:26:54.7578150Z 347c41648f51: Verifying Checksum 2024-05-19T18:26:54.7578580Z 347c41648f51: Download complete 2024-05-19T18:26:59.2683531Z 00953247955f: Verifying Checksum 2024-05-19T18:26:59.2684109Z 00953247955f: Download complete 2024-05-19T18:26:59.3500625Z df27b4a5ce7c: Download complete 2024-05-19T18:26:59.4341311Z acb7fcd74bf2: Download complete 2024-05-19T18:26:59.9088727Z 5bd6ac51ef5d: Verifying Checksum 2024-05-19T18:26:59.9089282Z 5bd6ac51ef5d: Download complete 2024-05-19T18:26:59.9949628Z 1795bce633b0: Verifying Checksum 2024-05-19T18:26:59.9950052Z 1795bce633b0: Download complete 2024-05-19T18:27:00.0883947Z e41de6016f3c: Verifying Checksum 2024-05-19T18:27:00.0884380Z e41de6016f3c: Download complete 2024-05-19T18:27:00.3586182Z 458f5f2af8cf: Verifying Checksum 2024-05-19T18:27:00.3586731Z 458f5f2af8cf: Download complete 2024-05-19T18:27:00.4387286Z ef99a2a9cffb: Verifying Checksum 2024-05-19T18:27:00.5403783Z 1ba8d08d5876: Verifying Checksum 2024-05-19T18:27:00.5404326Z 1ba8d08d5876: Download complete 2024-05-19T18:27:00.6247673Z 0c67ae0247cf: Verifying Checksum 2024-05-19T18:27:00.6248116Z 0c67ae0247cf: Download complete 2024-05-19T18:27:04.8106128Z ebfec18059b9: Pull complete 2024-05-19T18:27:05.0471619Z 533b4aebf169: Pull complete 2024-05-19T18:27:05.2835046Z 9dd75d06a091: Pull complete 2024-05-19T18:27:05.3805492Z 1b57ce94cad9: Verifying Checksum 2024-05-19T18:27:05.3806109Z 1b57ce94cad9: Download complete 2024-05-19T18:27:05.4678476Z a6b88127ee12: Verifying Checksum 2024-05-19T18:27:05.4679143Z a6b88127ee12: Download complete 2024-05-19T18:27:05.5251952Z 30bfca4dd349: Pull complete 2024-05-19T18:27:05.5626467Z 7b0cf0b8be50: Verifying Checksum 2024-05-19T18:27:05.5627133Z 7b0cf0b8be50: Download complete 2024-05-19T18:27:06.0719319Z 044061f29c13: Verifying Checksum 2024-05-19T18:27:06.0719902Z 044061f29c13: Download complete 2024-05-19T18:27:06.1786325Z f21869a483d4: Download complete 2024-05-19T18:27:06.2536540Z a155d9e6e7e7: Download complete 2024-05-19T18:27:06.3365874Z 6028f518cba8: Verifying Checksum 2024-05-19T18:27:06.3366437Z 6028f518cba8: Download complete 2024-05-19T18:27:06.4151806Z 9f0e16ae766e: Download complete 2024-05-19T18:27:11.6283186Z 7cb10a6ae8df: Verifying Checksum 2024-05-19T18:27:11.6283653Z 7cb10a6ae8df: Download complete 2024-05-19T18:27:11.7058736Z a78a7e442565: Download complete 2024-05-19T18:27:11.7861992Z df585564bf4c: Verifying Checksum 2024-05-19T18:27:11.7862519Z df585564bf4c: Download complete 2024-05-19T18:27:11.9233444Z f125619c4712: Verifying Checksum 2024-05-19T18:27:11.9233977Z f125619c4712: Download complete 2024-05-19T18:27:12.0126037Z 4cc671ff8870: Verifying Checksum 2024-05-19T18:27:12.0126526Z 4cc671ff8870: Download complete 2024-05-19T18:27:12.1236107Z e1d3209e04ae: Download complete 2024-05-19T18:27:12.1964789Z 322e57f70842: Verifying Checksum 2024-05-19T18:27:12.2742069Z 322e57f70842: Download complete 2024-05-19T18:27:12.2742588Z d58f09a95d3e: Verifying Checksum 2024-05-19T18:27:12.2742989Z d58f09a95d3e: Download complete 2024-05-19T18:27:12.4171677Z f611e28aa59a: Verifying Checksum 2024-05-19T18:27:12.4172340Z f611e28aa59a: Download complete 2024-05-19T18:27:12.5024247Z ff67e6bd8e23: Verifying Checksum 2024-05-19T18:27:12.5024927Z ff67e6bd8e23: Download complete 2024-05-19T18:27:13.1180825Z d2b06b7f79e8: Verifying Checksum 2024-05-19T18:27:13.1181495Z d2b06b7f79e8: Download complete 2024-05-19T18:27:13.2048975Z c27bc13ce9d4: Download complete 2024-05-19T18:27:40.0655809Z 26d45b11322a: Verifying Checksum 2024-05-19T18:27:40.0656806Z 26d45b11322a: Download complete 2024-05-19T18:27:40.1708957Z 400e9c337687: Download complete 2024-05-19T18:27:40.2491825Z 914d4657be1b: Download complete 2024-05-19T18:27:41.9097359Z ecc1d0285d28: Verifying Checksum 2024-05-19T18:27:41.9099072Z ecc1d0285d28: Download complete 2024-05-19T18:27:42.0135221Z b882dcc19062: Verifying Checksum 2024-05-19T18:27:42.0135844Z b882dcc19062: Download complete 2024-05-19T18:27:45.1487606Z 8680ed64cc19: Verifying Checksum 2024-05-19T18:27:45.1488123Z 8680ed64cc19: Download complete 2024-05-19T18:28:07.1984018Z 3ebf7de5956e: Verifying Checksum 2024-05-19T18:28:07.1984495Z 3ebf7de5956e: Download complete 2024-05-19T18:28:24.0520414Z 1b57ce94cad9: Pull complete 2024-05-19T18:28:24.2727412Z 9ee6bdb31195: Pull complete 2024-05-19T18:28:24.4980941Z 8af83a173aa3: Pull complete 2024-05-19T18:28:29.5753750Z f8281676bd20: Pull complete 2024-05-19T18:28:29.7975883Z 748e3cfdd8d6: Pull complete 2024-05-19T18:28:29.9663370Z 5fa8d379f04e: Pull complete 2024-05-19T18:28:30.1652242Z a95b9f41fb5b: Pull complete 2024-05-19T18:28:31.9741041Z 8b6d0b819d19: Pull complete 2024-05-19T18:28:32.2159802Z c61996e73399: Pull complete 2024-05-19T18:28:32.3936596Z 9e69dcd7661c: Pull complete 2024-05-19T18:28:32.5401747Z cfcaf2eb8249: Pull complete 2024-05-19T18:29:09.8083532Z 7cb10a6ae8df: Pull complete 2024-05-19T18:29:10.0199511Z 07fa87a43fb8: Pull complete 2024-05-19T18:29:10.1676697Z f16e94b04801: Pull complete 2024-05-19T18:29:10.3177690Z c857065196d6: Pull complete 2024-05-19T18:29:10.4631627Z 9d5c38cad0c9: Pull complete 2024-05-19T18:29:10.7081069Z 925ac0c7ff93: Pull complete 2024-05-19T18:29:13.8048556Z cfbf8bf4873d: Pull complete 2024-05-19T18:29:13.9291937Z 01914a189c0b: Pull complete 2024-05-19T18:29:14.1054599Z 3c2451fbe84c: Pull complete 2024-05-19T18:29:14.3065248Z aea8a2a73b72: Pull complete 2024-05-19T18:29:14.4987830Z c26f348bfda1: Pull complete 2024-05-19T18:29:14.6615059Z 347c41648f51: Pull complete 2024-05-19T18:29:20.5271987Z 00953247955f: Pull complete 2024-05-19T18:29:20.7158932Z df27b4a5ce7c: Pull complete 2024-05-19T18:29:20.9215408Z acb7fcd74bf2: Pull complete 2024-05-19T18:29:21.7882093Z 5bd6ac51ef5d: Pull complete 2024-05-19T18:29:21.9941876Z 1795bce633b0: Pull complete 2024-05-19T18:29:22.1968105Z e41de6016f3c: Pull complete 2024-05-19T18:29:22.6335603Z 458f5f2af8cf: Pull complete 2024-05-19T18:29:22.8634523Z ef99a2a9cffb: Pull complete 2024-05-19T18:29:23.3195699Z 1ba8d08d5876: Pull complete 2024-05-19T18:29:23.5378244Z 0c67ae0247cf: Pull complete 2024-05-19T18:30:00.6117088Z 3ebf7de5956e: Pull complete 2024-05-19T18:30:00.8390548Z a6b88127ee12: Pull complete 2024-05-19T18:30:01.0651273Z 7b0cf0b8be50: Pull complete 2024-05-19T18:30:02.3541748Z 044061f29c13: Pull complete 2024-05-19T18:30:02.5740910Z f21869a483d4: Pull complete 2024-05-19T18:30:02.7985197Z a155d9e6e7e7: Pull complete 2024-05-19T18:30:03.2642527Z 6028f518cba8: Pull complete 2024-05-19T18:30:03.4944288Z 9f0e16ae766e: Pull complete 2024-05-19T18:30:32.8393362Z 26d45b11322a: Pull complete 2024-05-19T18:30:33.0708707Z a78a7e442565: Pull complete 2024-05-19T18:30:33.3091588Z df585564bf4c: Pull complete 2024-05-19T18:30:33.6513804Z f125619c4712: Pull complete 2024-05-19T18:30:33.8697416Z 4cc671ff8870: Pull complete 2024-05-19T18:30:34.0962517Z e1d3209e04ae: Pull complete 2024-05-19T18:30:34.3182893Z 322e57f70842: Pull complete 2024-05-19T18:30:34.5425084Z d58f09a95d3e: Pull complete 2024-05-19T18:30:35.1950265Z f611e28aa59a: Pull complete 2024-05-19T18:30:35.4348386Z ff67e6bd8e23: Pull complete 2024-05-19T18:30:37.1315412Z d2b06b7f79e8: Pull complete 2024-05-19T18:30:37.3557896Z c27bc13ce9d4: Pull complete 2024-05-19T18:30:51.7814534Z 8680ed64cc19: Pull complete 2024-05-19T18:30:52.0057385Z 400e9c337687: Pull complete 2024-05-19T18:30:52.2337442Z 914d4657be1b: Pull complete 2024-05-19T18:30:53.0716908Z ecc1d0285d28: Pull complete 2024-05-19T18:30:53.3159315Z b882dcc19062: Pull complete 2024-05-19T18:30:53.4206509Z Digest: sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T18:30:53.4648865Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:30:53.4910285Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:30:53.4953193Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:30:53.4954141Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-05-19T18:30:53.4962492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:30:53.4963001Z env: 2024-05-19T18:30:53.4963281Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:30:53.4963639Z ##[endgroup] 2024-05-19T18:30:53.5119216Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-05-19T18:30:53.5119773Z with: 2024-05-19T18:30:53.5120071Z driver-version: 550.54.15 2024-05-19T18:30:53.5120417Z env: 2024-05-19T18:30:53.5120697Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:30:53.5121052Z ##[endgroup] 2024-05-19T18:30:53.5249553Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:30:53.5250113Z with: 2024-05-19T18:30:53.5250407Z timeout_minutes: 10 2024-05-19T18:30:53.5250739Z max_attempts: 3 2024-05-19T18:30:53.5281571Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-05-19T18:30:53.5313028Z retry_wait_seconds: 10 2024-05-19T18:30:53.5313400Z polling_interval_seconds: 1 2024-05-19T18:30:53.5313775Z warning_on_retry: true 2024-05-19T18:30:53.5314131Z continue_on_error: false 2024-05-19T18:30:53.5314471Z env: 2024-05-19T18:30:53.5314749Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:30:53.5315103Z DRIVER_VERSION: 550.54.15 2024-05-19T18:30:53.5315455Z ##[endgroup] 2024-05-19T18:30:53.5838372Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-05-19T18:30:53.5839144Z + pre_install_nvidia_driver_amzn2 2024-05-19T18:30:53.5842175Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-05-19T18:30:53.8907211Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:30:53.9307479Z No Match for argument: nvidia-driver-latest-dkms 2024-05-19T18:30:53.9552422Z No Packages marked for removal 2024-05-19T18:30:53.9685894Z + install_nvidia_driver_common 2024-05-19T18:30:53.9688769Z + echo 'Before installing NVIDIA driver' 2024-05-19T18:30:53.9689660Z + lspci 2024-05-19T18:30:53.9690012Z Before installing NVIDIA driver 2024-05-19T18:30:54.0550566Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:30:54.0551852Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:30:54.0552797Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:30:54.0553597Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:30:54.0554368Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:30:54.0555114Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:30:54.0555836Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:30:54.0556617Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:30:54.0557207Z + lsmod 2024-05-19T18:30:54.0564534Z Module Size Used by 2024-05-19T18:30:54.0565262Z backlight 16384 0 2024-05-19T18:30:54.0565902Z xt_conntrack 16384 1 2024-05-19T18:30:54.0566511Z ipt_MASQUERADE 16384 1 2024-05-19T18:30:54.0567133Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:30:54.0567805Z nf_conntrack_netlink 49152 0 2024-05-19T18:30:54.0568320Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:30:54.0568798Z xfrm_user 45056 1 2024-05-19T18:30:54.0569230Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:30:54.0569894Z iptable_nat 16384 1 2024-05-19T18:30:54.0570296Z nf_conntrack_ipv4 16384 3 2024-05-19T18:30:54.0570746Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:30:54.0571234Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:30:54.0571908Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:30:54.0573428Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:30:54.0574452Z xt_addrtype 16384 2 2024-05-19T18:30:54.0575034Z iptable_filter 16384 1 2024-05-19T18:30:54.0575642Z br_netfilter 24576 0 2024-05-19T18:30:54.0576275Z bridge 172032 1 br_netfilter 2024-05-19T18:30:54.0576966Z stp 16384 1 bridge 2024-05-19T18:30:54.0577603Z llc 16384 2 bridge,stp 2024-05-19T18:30:54.0578219Z overlay 86016 0 2024-05-19T18:30:54.0578766Z sunrpc 393216 1 2024-05-19T18:30:54.0579296Z dm_mirror 28672 0 2024-05-19T18:30:54.0579838Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:30:54.0580493Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:30:54.0581207Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:30:54.0581837Z dax 69632 1 dm_mod 2024-05-19T18:30:54.0582409Z crc32_pclmul 16384 0 2024-05-19T18:30:54.0582928Z ghash_clmulni_intel 16384 0 2024-05-19T18:30:54.0583456Z pcbc 16384 0 2024-05-19T18:30:54.0583953Z aesni_intel 188416 0 2024-05-19T18:30:54.0584480Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:30:54.0585080Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:30:54.0585658Z mousedev 24576 0 2024-05-19T18:30:54.0586078Z glue_helper 16384 1 aesni_intel 2024-05-19T18:30:54.0586680Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:30:54.0587347Z evdev 20480 3 2024-05-19T18:30:54.0587754Z psmouse 32768 0 2024-05-19T18:30:54.0588146Z button 16384 0 2024-05-19T18:30:54.0588507Z ena 139264 0 2024-05-19T18:30:54.0588886Z ptp 20480 1 ena 2024-05-19T18:30:54.0589287Z pps_core 20480 1 ptp 2024-05-19T18:30:54.0589685Z crc32c_intel 24576 0 2024-05-19T18:30:54.0590050Z autofs4 49152 2 2024-05-19T18:30:54.0590559Z + modinfo nvidia 2024-05-19T18:30:54.0591364Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:30:54.0592059Z firmware: nvidia/535.54.03/gsp_tu10x.bin 2024-05-19T18:30:54.0592537Z firmware: nvidia/535.54.03/gsp_ga10x.bin 2024-05-19T18:30:54.0593029Z alias: char-major-195-* 2024-05-19T18:30:54.0593414Z version: 535.54.03 2024-05-19T18:30:54.0593765Z supported: external 2024-05-19T18:30:54.0594102Z license: NVIDIA 2024-05-19T18:30:54.0594470Z srcversion: EA9C7EF32617E104C8240C4 2024-05-19T18:30:54.0594949Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:30:54.0595443Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:30:54.0595953Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:30:54.0596433Z depends: i2c-core,drm 2024-05-19T18:30:54.0596798Z retpoline: Y 2024-05-19T18:30:54.0597103Z name: nvidia 2024-05-19T18:30:54.0597919Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:30:54.0598593Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:30:54.0599236Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:30:54.0599820Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:30:54.0600270Z parm: NVreg_RmLogonRC:int 2024-05-19T18:30:54.0600706Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:30:54.0601159Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:30:54.0601694Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:30:54.0602149Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:30:54.0602682Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:30:54.0603237Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:30:54.0603719Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:30:54.0604153Z parm: NVreg_EnableMSI:int 2024-05-19T18:30:54.0604572Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:30:54.0605031Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:30:54.0605561Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:30:54.0606141Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:30:54.0606706Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:30:54.0607312Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:30:54.0607920Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:30:54.0608550Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:30:54.0609147Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:30:54.0609634Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:30:54.0610171Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:30:54.0610726Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:30:54.0611229Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:30:54.0611693Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:30:54.0612181Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:30:54.0612655Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:30:54.0613106Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:30:54.0613601Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:30:54.0614126Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:30:54.0614606Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:30:54.0615104Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:30:54.0615580Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:30:54.0616081Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:30:54.0616565Z parm: NVreg_RmMsg:charp 2024-05-19T18:30:54.0616971Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:30:54.0617440Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:30:54.0617929Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:30:54.0618471Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:30:54.0618946Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:30:54.0619413Z parm: rm_firmware_active:charp 2024-05-19T18:30:54.0619842Z + HAS_NVIDIA_DRIVER=0 2024-05-19T18:30:54.0620228Z ++ command -v nvidia-smi 2024-05-19T18:30:54.0620619Z + '[' -x /usr/bin/nvidia-smi ']' 2024-05-19T18:30:54.0620990Z + set +e 2024-05-19T18:30:54.0621491Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-05-19T18:30:56.6292137Z + INSTALLED_DRIVER_VERSION=535.54.03 2024-05-19T18:30:56.6292771Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:30:56.6293266Z + '[' 0 -ne 0 ']' 2024-05-19T18:30:56.6293647Z + '[' 535.54.03 '!=' 550.54.15 ']' 2024-05-19T18:30:56.6294459Z + echo 'NVIDIA driver (535.54.03) has been installed, but we expect to have 550.54.15 instead. Continuing' 2024-05-19T18:30:56.6295197Z + set -e 2024-05-19T18:30:56.6295495Z + '[' 0 -eq 0 ']' 2024-05-19T18:30:56.6295859Z + '[' amzn2 '!=' ubuntu20.04 ']' 2024-05-19T18:30:56.6296370Z + sudo yum groupinstall -y 'Development Tools' 2024-05-19T18:30:56.6297121Z NVIDIA driver (535.54.03) has been installed, but we expect to have 550.54.15 instead. Continuing 2024-05-19T18:30:56.9324571Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:30:58.0868507Z Maybe run: yum groups mark install (see man yum) 2024-05-19T18:30:58.0870147Z No packages in any requested group available to install or update 2024-05-19T18:30:58.1776269Z ++ uname -r 2024-05-19T18:30:58.1783489Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.336-257.562.amzn2.x86_64' 2024-05-19T18:30:58.5008060Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:30:59.9668169Z Package kernel-devel-4.14.336-257.562.amzn2.x86_64 already installed and latest version 2024-05-19T18:30:59.9668954Z Nothing to do 2024-05-19T18:31:00.1145312Z + sudo modprobe backlight 2024-05-19T18:31:00.1530154Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-550.54.15.run 2024-05-19T18:31:03.2797612Z + set +e 2024-05-19T18:31:03.2798681Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm 2024-05-19T18:31:04.2306801Z Verifying archive integrity... OK 2024-05-19T18:31:06.7850660Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 550.54.15........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ 2024-05-19T18:31:06.9694933Z 2024-05-19T18:31:06.9696410Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver. 2024-05-19T18:31:06.9697236Z 2024-05-19T18:31:21.1498823Z 2024-05-19T18:31:21.1502420Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system. If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver. 2024-05-19T18:31:21.1504551Z 2024-05-19T18:31:21.1507891Z 2024-05-19T18:31:21.1509774Z WARNING: This NVIDIA driver package includes Vulkan components, but no Vulkan ICD loader was detected on this system. The NVIDIA Vulkan ICD will not function without the loader. Most distributions package the Vulkan loader; try installing the "vulkan-loader", "vulkan-icd-loader", or "libvulkan1" package. 2024-05-19T18:31:21.1511476Z 2024-05-19T18:31:29.4539063Z + NVIDIA_INSTALLATION_STATUS=0 2024-05-19T18:31:29.4539780Z + RESET_GPU=0 2024-05-19T18:31:29.4540257Z + '[' 0 -ne 0 ']' 2024-05-19T18:31:29.4540699Z ++ command -v nvidia-smi 2024-05-19T18:31:29.4541914Z + '[' -x /usr/bin/nvidia-smi ']' 2024-05-19T18:31:29.4544219Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-05-19T18:31:31.6200083Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-05-19T18:31:31.6200762Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:31:31.6201640Z + '[' 0 -ne 0 ']' 2024-05-19T18:31:31.6202033Z + '[' 0 -eq 1 ']' 2024-05-19T18:31:31.6202413Z + sudo rm -fv /tmp/nvidia_driver 2024-05-19T18:31:31.6568913Z removed '/tmp/nvidia_driver' 2024-05-19T18:31:31.6583667Z + set -e 2024-05-19T18:31:31.6586152Z + post_install_nvidia_driver_common 2024-05-19T18:31:31.6588589Z + sudo modprobe nvidia 2024-05-19T18:31:31.7053528Z + echo 'After installing NVIDIA driver' 2024-05-19T18:31:31.7054222Z + lspci 2024-05-19T18:31:31.7055001Z After installing NVIDIA driver 2024-05-19T18:31:31.7133798Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-05-19T18:31:31.7134611Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-05-19T18:31:31.7135542Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-05-19T18:31:31.7136547Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-05-19T18:31:31.7137605Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-05-19T18:31:31.7138490Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-05-19T18:31:31.7139232Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-05-19T18:31:31.7140027Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-05-19T18:31:31.7140625Z + lsmod 2024-05-19T18:31:31.7145750Z Module Size Used by 2024-05-19T18:31:31.7146363Z nvidia_uvm 4599808 0 2024-05-19T18:31:31.7146965Z nvidia 53989376 1 nvidia_uvm 2024-05-19T18:31:31.7147509Z drm 421888 1 nvidia 2024-05-19T18:31:31.7147933Z i2c_core 77824 2 nvidia,drm 2024-05-19T18:31:31.7148365Z backlight 16384 0 2024-05-19T18:31:31.7148749Z xt_conntrack 16384 1 2024-05-19T18:31:31.7149134Z ipt_MASQUERADE 16384 1 2024-05-19T18:31:31.7149569Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-05-19T18:31:31.7150056Z nf_conntrack_netlink 49152 0 2024-05-19T18:31:31.7150511Z nfnetlink 16384 2 nf_conntrack_netlink 2024-05-19T18:31:31.7150992Z xfrm_user 45056 1 2024-05-19T18:31:31.7151484Z xfrm_algo 16384 1 xfrm_user 2024-05-19T18:31:31.7151913Z iptable_nat 16384 1 2024-05-19T18:31:31.7152300Z nf_conntrack_ipv4 16384 3 2024-05-19T18:31:31.7152743Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-05-19T18:31:31.7153360Z nf_nat_ipv4 16384 1 iptable_nat 2024-05-19T18:31:31.7154010Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-05-19T18:31:31.7155023Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-05-19T18:31:31.7155876Z xt_addrtype 16384 2 2024-05-19T18:31:31.7156364Z iptable_filter 16384 1 2024-05-19T18:31:31.7156917Z br_netfilter 24576 0 2024-05-19T18:31:31.7157460Z bridge 172032 1 br_netfilter 2024-05-19T18:31:31.7158235Z stp 16384 1 bridge 2024-05-19T18:31:31.7158922Z llc 16384 2 bridge,stp 2024-05-19T18:31:31.7159349Z overlay 86016 0 2024-05-19T18:31:31.7159726Z sunrpc 393216 1 2024-05-19T18:31:31.7160104Z dm_mirror 28672 0 2024-05-19T18:31:31.7160508Z dm_region_hash 20480 1 dm_mirror 2024-05-19T18:31:31.7160999Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-05-19T18:31:31.7161518Z dm_mod 143360 2 dm_log,dm_mirror 2024-05-19T18:31:31.7161985Z dax 69632 1 dm_mod 2024-05-19T18:31:31.7162401Z crc32_pclmul 16384 0 2024-05-19T18:31:31.7162781Z ghash_clmulni_intel 16384 0 2024-05-19T18:31:31.7163166Z pcbc 16384 0 2024-05-19T18:31:31.7163541Z aesni_intel 188416 0 2024-05-19T18:31:31.7163932Z aes_x86_64 20480 1 aesni_intel 2024-05-19T18:31:31.7164387Z crypto_simd 16384 1 aesni_intel 2024-05-19T18:31:31.7164820Z mousedev 24576 0 2024-05-19T18:31:31.7165240Z glue_helper 16384 1 aesni_intel 2024-05-19T18:31:31.7165789Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-05-19T18:31:31.7166336Z evdev 20480 3 2024-05-19T18:31:31.7166719Z psmouse 32768 0 2024-05-19T18:31:31.7167091Z button 16384 0 2024-05-19T18:31:31.7167449Z ena 139264 0 2024-05-19T18:31:31.7167825Z ptp 20480 1 ena 2024-05-19T18:31:31.7168227Z pps_core 20480 1 ptp 2024-05-19T18:31:31.7168707Z crc32c_intel 24576 0 2024-05-19T18:31:31.7169088Z autofs4 49152 2 2024-05-19T18:31:31.7169455Z + modinfo nvidia 2024-05-19T18:31:31.7170115Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-05-19T18:31:31.7170808Z alias: char-major-195-* 2024-05-19T18:31:31.7171201Z version: 550.54.15 2024-05-19T18:31:31.7171555Z supported: external 2024-05-19T18:31:31.7171899Z license: NVIDIA 2024-05-19T18:31:31.7172282Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-05-19T18:31:31.7172771Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-05-19T18:31:31.7173238Z srcversion: 833721318DA517F0C2FEC97 2024-05-19T18:31:31.7173706Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-05-19T18:31:31.7174205Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-05-19T18:31:31.7174713Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-05-19T18:31:31.7175199Z depends: i2c-core,drm 2024-05-19T18:31:31.7175577Z retpoline: Y 2024-05-19T18:31:31.7175883Z name: nvidia 2024-05-19T18:31:31.7176442Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-05-19T18:31:31.7177106Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-05-19T18:31:31.7177755Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-05-19T18:31:31.7178362Z parm: NVreg_ResmanDebugLevel:int 2024-05-19T18:31:31.7178818Z parm: NVreg_RmLogonRC:int 2024-05-19T18:31:31.7179260Z parm: NVreg_ModifyDeviceFiles:int 2024-05-19T18:31:31.7179718Z parm: NVreg_DeviceFileUID:int 2024-05-19T18:31:31.7180158Z parm: NVreg_DeviceFileGID:int 2024-05-19T18:31:31.7180604Z parm: NVreg_DeviceFileMode:int 2024-05-19T18:31:31.7181338Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-05-19T18:31:31.7181903Z parm: NVreg_UsePageAttributeTable:int 2024-05-19T18:31:31.7182391Z parm: NVreg_EnablePCIeGen3:int 2024-05-19T18:31:31.7182827Z parm: NVreg_EnableMSI:int 2024-05-19T18:31:31.7183246Z parm: NVreg_TCEBypassMode:int 2024-05-19T18:31:31.7183707Z parm: NVreg_EnableStreamMemOPs:int 2024-05-19T18:31:31.7184249Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-05-19T18:31:31.7184834Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-05-19T18:31:31.7185402Z parm: NVreg_EnableS0ixPowerManagement:int 2024-05-19T18:31:31.7186083Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-05-19T18:31:31.7186685Z parm: NVreg_DynamicPowerManagement:int 2024-05-19T18:31:31.7187305Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-05-19T18:31:31.7187905Z parm: NVreg_EnableGpuFirmware:int 2024-05-19T18:31:31.7188392Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-05-19T18:31:31.7188929Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-05-19T18:31:31.7189490Z parm: NVreg_EnableUserNUMAManagement:int 2024-05-19T18:31:31.7189988Z parm: NVreg_MemoryPoolSize:int 2024-05-19T18:31:31.7190457Z parm: NVreg_KMallocHeapMaxSize:int 2024-05-19T18:31:31.7190948Z parm: NVreg_VMallocHeapMaxSize:int 2024-05-19T18:31:31.7191553Z parm: NVreg_IgnoreMMIOCheck:int 2024-05-19T18:31:31.7192005Z parm: NVreg_NvLinkDisable:int 2024-05-19T18:31:31.7192521Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-05-19T18:31:31.7193064Z parm: NVreg_RegisterPCIDriver:int 2024-05-19T18:31:31.7193554Z parm: NVreg_EnableResizableBar:int 2024-05-19T18:31:31.7194046Z parm: NVreg_EnableDbgBreakpoint:int 2024-05-19T18:31:31.7194557Z parm: NVreg_EnableNonblockingOpen:int 2024-05-19T18:31:31.7195052Z parm: NVreg_RegistryDwords:charp 2024-05-19T18:31:31.7195569Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-05-19T18:31:31.7196051Z parm: NVreg_RmMsg:charp 2024-05-19T18:31:31.7196543Z parm: NVreg_GpuBlacklist:charp 2024-05-19T18:31:31.7197078Z parm: NVreg_TemporaryFilePath:charp 2024-05-19T18:31:31.7197765Z parm: NVreg_ExcludedGpus:charp 2024-05-19T18:31:31.7198271Z parm: NVreg_DmaRemapPeerMmio:int 2024-05-19T18:31:31.7198795Z parm: NVreg_RmNvlinkBandwidth:charp 2024-05-19T18:31:31.7199284Z parm: NVreg_ImexChannelCount:int 2024-05-19T18:31:31.7199737Z parm: rm_firmware_active:charp 2024-05-19T18:31:31.7200159Z + set +e 2024-05-19T18:31:31.7200484Z + nvidia-smi 2024-05-19T18:31:33.2704008Z Sun May 19 18:31:33 2024 2024-05-19T18:31:33.2704967Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:31:33.2705801Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T18:31:33.2706620Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:31:33.2707455Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T18:31:33.2708359Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T18:31:33.2709058Z | | | MIG M. | 2024-05-19T18:31:33.2709597Z |=========================================+========================+======================| 2024-05-19T18:31:33.2842747Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T18:31:33.2843613Z | 0% 28C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-05-19T18:31:33.2844255Z | | | N/A | 2024-05-19T18:31:33.2845155Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T18:31:33.2845804Z 2024-05-19T18:31:33.2846474Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:31:33.2847110Z | Processes: | 2024-05-19T18:31:33.2847821Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T18:31:33.2848516Z | ID ID Usage | 2024-05-19T18:31:33.2849372Z |=========================================================================================| 2024-05-19T18:31:33.2850024Z | No running processes found | 2024-05-19T18:31:33.2850786Z +-----------------------------------------------------------------------------------------+ 2024-05-19T18:31:33.8866378Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T18:31:35.4473346Z NVIDIA A10G 2024-05-19T18:31:35.8910658Z + NVIDIA_SMI_STATUS=0 2024-05-19T18:31:35.8911405Z + '[' 0 -eq 0 ']' 2024-05-19T18:31:35.8911828Z + echo 'INFO: Ignoring allowed status 0' 2024-05-19T18:31:35.8912267Z + set -e 2024-05-19T18:31:35.8912583Z INFO: Ignoring allowed status 0 2024-05-19T18:31:35.8915149Z == Installing nvidia container toolkit for amzn2 == 2024-05-19T18:31:35.8916712Z + sudo yum install -y yum-utils 2024-05-19T18:31:36.1944404Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:31:37.6667749Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-05-19T18:31:37.6668472Z Nothing to do 2024-05-19T18:31:37.8147790Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:31:38.1384253Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:31:38.1665874Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-05-19T18:31:38.1668764Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:31:38.1669764Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-05-19T18:31:38.1783685Z + sudo yum install -y nvidia-docker2 2024-05-19T18:31:38.4780121Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-05-19T18:31:39.9531547Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-05-19T18:31:39.9532245Z Nothing to do 2024-05-19T18:31:40.1044280Z + sudo systemctl restart docker 2024-05-19T18:32:08.6542562Z Command completed after 1 attempt(s). 2024-05-19T18:32:08.6604397Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:32:08.6605177Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-05-19T18:32:08.6605883Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-05-19T18:32:08.6606549Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-05-19T18:32:08.6614464Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:08.6614972Z env: 2024-05-19T18:32:08.6615256Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:08.6615714Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:08.6616191Z ##[endgroup] 2024-05-19T18:32:08.9941095Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:32:09.3981305Z Collecting psutil==5.9.1 2024-05-19T18:32:09.4205927Z Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-05-19T18:32:09.4690831Z Collecting nvidia-ml-py==11.525.84 2024-05-19T18:32:09.4758895Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-05-19T18:32:09.5148736Z Installing collected packages: psutil, nvidia-ml-py 2024-05-19T18:32:09.6696585Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-05-19T18:32:09.8651572Z Prepare all required actions 2024-05-19T18:32:09.8652082Z Getting action download info 2024-05-19T18:32:09.9703511Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-05-19T18:32:10.1692250Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-05-19T18:32:10.3084160Z ##[group]Run ./.github/actions/download-build-artifacts 2024-05-19T18:32:10.3084650Z with: 2024-05-19T18:32:10.3084995Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:32:10.3085582Z s3-bucket: gha-artifacts 2024-05-19T18:32:10.3085924Z env: 2024-05-19T18:32:10.3086200Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:10.3086670Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:10.3087156Z ##[endgroup] 2024-05-19T18:32:11.3282037Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:32:11.3282490Z with: 2024-05-19T18:32:11.3282874Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:32:11.3283340Z s3-bucket: gha-artifacts 2024-05-19T18:32:11.3283698Z region: us-east-1 2024-05-19T18:32:11.3284008Z env: 2024-05-19T18:32:11.3284295Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:11.3284743Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:11.3285233Z ##[endgroup] 2024-05-19T18:32:11.7557911Z (node:28843) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:32:11.7558717Z 2024-05-19T18:32:11.7558986Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:32:11.7559798Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:32:11.7560694Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:32:11.8431373Z Found 1 objects with prefix pytorch/pytorch/9148947323/linux-focal-cuda12.4-py3.10-gcc9-sm86/ 2024-05-19T18:32:11.8432481Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:32:19.3215207Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-05-19T18:32:19.3222205Z Artifact download has finished successfully 2024-05-19T18:32:19.3393173Z ##[group]Run unzip -o artifacts.zip 2024-05-19T18:32:19.3393612Z unzip -o artifacts.zip 2024-05-19T18:32:19.3401818Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:19.3402332Z env: 2024-05-19T18:32:19.3402617Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:19.3403092Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:19.3403584Z ##[endgroup] 2024-05-19T18:32:19.3447844Z Archive: artifacts.zip 2024-05-19T18:32:19.3449439Z creating: dist/ 2024-05-19T18:32:21.3186884Z inflating: dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:32:21.3188232Z creating: build/custom_test_artifacts/ 2024-05-19T18:32:21.3189401Z creating: build/custom_test_artifacts/custom-op-build/ 2024-05-19T18:32:21.3190295Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-05-19T18:32:21.3191230Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:32:21.3193257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:32:21.3194206Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-05-19T18:32:21.3195139Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:32:21.3196143Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:32:21.3197110Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:32:21.3198623Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:32:21.3199945Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:32:21.3200977Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:32:21.3201975Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:32:21.3203141Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:32:21.3204313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:32:21.3205592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:32:21.3206744Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:32:21.3207911Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:32:21.3226364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:32:21.3227436Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:32:21.3228453Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:32:21.3246695Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:32:21.3286150Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:32:21.3287645Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:32:21.3331386Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:32:21.3332804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:32:21.3334224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:32:21.3335694Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:32:21.3337116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:32:21.3338502Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:32:21.3339890Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:32:21.3341271Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:32:21.3342609Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:32:21.3343894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:32:21.3345128Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:32:21.3346347Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:32:21.3347573Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:32:21.3348761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:32:21.3349970Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:32:21.3413578Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:32:21.3484954Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:32:21.3486174Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:32:21.3487178Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:32:21.3488060Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:32:21.3489129Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-05-19T18:32:21.3490067Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-05-19T18:32:21.3491101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-05-19T18:32:21.3492251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-05-19T18:32:21.3493348Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-05-19T18:32:21.3494385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-05-19T18:32:21.3495460Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-05-19T18:32:21.3496530Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-05-19T18:32:21.3497602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-05-19T18:32:21.3498683Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-05-19T18:32:21.3499740Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-05-19T18:32:21.3512214Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-05-19T18:32:21.3641086Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-05-19T18:32:21.3642098Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-05-19T18:32:21.3643194Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-05-19T18:32:21.3644415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-05-19T18:32:21.3645581Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-05-19T18:32:21.3646675Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-05-19T18:32:21.3647803Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-05-19T18:32:21.3648946Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-05-19T18:32:21.3650078Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-05-19T18:32:21.3651204Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-05-19T18:32:21.3652321Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-05-19T18:32:21.3666287Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-05-19T18:32:21.3746845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-05-19T18:32:21.3749219Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:32:21.3750561Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:32:21.3751743Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-05-19T18:32:21.3752652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-05-19T18:32:21.3753545Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-05-19T18:32:21.3754453Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-05-19T18:32:21.3755287Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-05-19T18:32:21.3756063Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-05-19T18:32:21.3756943Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-05-19T18:32:21.3862136Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-05-19T18:32:21.3923200Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-05-19T18:32:21.3923941Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-05-19T18:32:21.3924639Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-05-19T18:32:21.3925452Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:32:21.3931569Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:32:21.3932495Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-05-19T18:32:21.3933429Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:32:21.3934412Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:32:21.3935392Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:32:21.3936477Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:32:21.3937575Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:32:21.3938567Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:32:21.3939564Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:32:21.3940697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:32:21.3941833Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:32:21.3942951Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:32:21.3944086Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:32:21.3945227Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:32:21.3946377Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:32:21.3947395Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:32:21.3948376Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:32:21.3988031Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:32:21.4026712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:32:21.4028168Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:32:21.4072449Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:32:21.4073852Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:32:21.4075409Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:32:21.4076890Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:32:21.4078572Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:32:21.4079982Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:32:21.4081491Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:32:21.4082865Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:32:21.4084210Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:32:21.4085480Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:32:21.4086706Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:32:21.4087907Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:32:21.4089125Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:32:21.4090314Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:32:21.4091519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:32:21.4156523Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:32:21.4228633Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:32:21.4230450Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:32:21.4232003Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:32:21.4233263Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:32:21.4234579Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-05-19T18:32:21.4235989Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-05-19T18:32:21.4237543Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-05-19T18:32:21.4239457Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-05-19T18:32:21.4241159Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-05-19T18:32:21.4242661Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-05-19T18:32:21.4244237Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-05-19T18:32:21.4245786Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-05-19T18:32:21.4247372Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-05-19T18:32:21.4248960Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-05-19T18:32:21.4250505Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-05-19T18:32:21.4258753Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-05-19T18:32:21.4321122Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-05-19T18:32:21.4322869Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:32:21.4324440Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:32:21.4325824Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-05-19T18:32:21.4327112Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-05-19T18:32:21.4328570Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-05-19T18:32:21.4329890Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-05-19T18:32:21.4331109Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-05-19T18:32:21.4332207Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-05-19T18:32:21.4333306Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-05-19T18:32:21.4378019Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-05-19T18:32:21.4379117Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-05-19T18:32:21.4380176Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-05-19T18:32:21.4381397Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-05-19T18:32:21.4386499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-05-19T18:32:21.4387964Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-05-19T18:32:21.4389390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-05-19T18:32:21.4390905Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-05-19T18:32:21.4392558Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-05-19T18:32:21.4394226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-05-19T18:32:21.4395932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-05-19T18:32:21.4397479Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-05-19T18:32:21.4399207Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-05-19T18:32:21.4401028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-05-19T18:32:21.4402790Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-05-19T18:32:21.4404532Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-05-19T18:32:21.4406255Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-05-19T18:32:21.4408007Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-05-19T18:32:21.4409778Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-05-19T18:32:21.4411316Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-05-19T18:32:21.4412882Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-05-19T18:32:21.4445348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-05-19T18:32:21.4485551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-05-19T18:32:21.4487856Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-05-19T18:32:21.4531899Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-05-19T18:32:21.4533976Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-05-19T18:32:21.4536183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-05-19T18:32:21.4538156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-05-19T18:32:21.4539615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-05-19T18:32:21.4552351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-05-19T18:32:21.4554039Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-05-19T18:32:21.4555706Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-05-19T18:32:21.4557133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-05-19T18:32:21.4560083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-05-19T18:32:21.4561458Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-05-19T18:32:21.4562750Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-05-19T18:32:21.4564066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-05-19T18:32:21.4565333Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-05-19T18:32:21.4566614Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-05-19T18:32:21.4632352Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-05-19T18:32:21.4704160Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-05-19T18:32:21.4706699Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-05-19T18:32:21.4708806Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-05-19T18:32:21.4710279Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-05-19T18:32:21.4711347Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-05-19T18:32:21.4712372Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-05-19T18:32:21.4713509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-05-19T18:32:21.4714776Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-05-19T18:32:21.4715986Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-05-19T18:32:21.4717133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-05-19T18:32:21.4718595Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-05-19T18:32:21.4719970Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-05-19T18:32:21.4721175Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-05-19T18:32:21.4722372Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-05-19T18:32:21.4723546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-05-19T18:32:21.4724802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-05-19T18:32:21.4832511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-05-19T18:32:21.4833698Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-05-19T18:32:21.4834901Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-05-19T18:32:21.4836215Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-05-19T18:32:21.4837481Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-05-19T18:32:21.4838788Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-05-19T18:32:21.4840045Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-05-19T18:32:21.4841358Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-05-19T18:32:21.4842603Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-05-19T18:32:21.4843850Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-05-19T18:32:21.4845083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-05-19T18:32:21.4856190Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-05-19T18:32:21.4909465Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-05-19T18:32:21.4911397Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-05-19T18:32:21.4913698Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-05-19T18:32:21.4915769Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-05-19T18:32:21.4917879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-05-19T18:32:21.4919807Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-05-19T18:32:21.4921026Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-05-19T18:32:21.4921938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-05-19T18:32:21.4922776Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-05-19T18:32:21.4923648Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-05-19T18:32:21.5014806Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-05-19T18:32:21.5055923Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-05-19T18:32:21.5056561Z creating: build/lib/ 2024-05-19T18:32:21.5141715Z inflating: build/lib/libprotobuf-lite.a 2024-05-19T18:32:21.5582926Z inflating: build/lib/libprotobuf.a 2024-05-19T18:32:21.5591947Z inflating: build/lib/libpthreadpool.a 2024-05-19T18:32:21.5599844Z inflating: build/lib/libcpuinfo.a 2024-05-19T18:32:21.5607320Z inflating: build/lib/libcpuinfo_internals.a 2024-05-19T18:32:21.5608059Z inflating: build/lib/libclog.a 2024-05-19T18:32:21.5625335Z inflating: build/lib/libnnpack.a 2024-05-19T18:32:21.5627338Z inflating: build/lib/libnnpack_reference_layers.a 2024-05-19T18:32:21.5690021Z inflating: build/lib/libgtest.a 2024-05-19T18:32:21.5762452Z inflating: build/lib/libbenchmark.a 2024-05-19T18:32:21.5824264Z inflating: build/lib/libasmjit.a 2024-05-19T18:32:21.5831409Z inflating: build/lib/libittnotify.a 2024-05-19T18:32:21.5858168Z inflating: build/lib/libtensorpipe_uv.a 2024-05-19T18:32:21.5980804Z inflating: build/lib/libgloo.a 2024-05-19T18:32:21.6002103Z inflating: build/lib/libfmt.a 2024-05-19T18:32:21.6090354Z inflating: build/lib/libc10.so 2024-05-19T18:32:21.6091817Z inflating: build/lib/libcaffe2_nvrtc.so 2024-05-19T18:32:21.6092301Z inflating: build/lib/libfoxi_loader.a 2024-05-19T18:32:21.6093296Z inflating: build/lib/libtorch_global_deps.so 2024-05-19T18:32:21.6106361Z inflating: build/lib/libqnnpack.a 2024-05-19T18:32:21.6595172Z inflating: build/lib/libprotoc.a 2024-05-19T18:32:21.6613421Z inflating: build/lib/libpytorch_qnnpack.a 2024-05-19T18:32:21.6630639Z inflating: build/lib/libgmock.a 2024-05-19T18:32:21.6631240Z inflating: build/lib/libgtest_main.a 2024-05-19T18:32:21.6631723Z inflating: build/lib/libbenchmark_main.a 2024-05-19T18:32:21.7004208Z inflating: build/lib/libgloo_cuda.a 2024-05-19T18:32:22.6795791Z inflating: build/lib/libdnnl.a 2024-05-19T18:32:22.7348793Z inflating: build/lib/libtensorpipe.a 2024-05-19T18:32:22.7401417Z inflating: build/lib/libc10_cuda.so 2024-05-19T18:32:22.7401890Z inflating: build/lib/libgmock_main.a 2024-05-19T18:32:22.8610928Z inflating: build/lib/libfbgemm.a 2024-05-19T18:32:22.8857476Z inflating: build/lib/libtensorpipe_cuda.a 2024-05-19T18:32:22.9357114Z inflating: build/lib/libkineto.a 2024-05-19T18:32:22.9394206Z inflating: build/lib/libcaffe2_protos.a 2024-05-19T18:32:22.9434897Z inflating: build/lib/libonnx_proto.a 2024-05-19T18:32:22.9613911Z inflating: build/lib/libXNNPACK.a 2024-05-19T18:32:23.0292470Z inflating: build/lib/libonnx.a 2024-05-19T18:32:25.3480067Z inflating: build/lib/libtorch_cpu.so 2024-05-19T18:32:25.3484400Z inflating: build/lib/libunbox_lib.a 2024-05-19T18:32:25.3487499Z inflating: build/lib/libshm.so 2024-05-19T18:32:27.3330622Z inflating: build/lib/libtorch_cuda.so 2024-05-19T18:32:27.3331297Z inflating: build/lib/libtorch.so 2024-05-19T18:32:27.3334559Z inflating: build/lib/libc10d_cuda_test.so 2024-05-19T18:32:28.1909098Z inflating: build/lib/libtorch_cuda_linalg.so 2024-05-19T18:32:28.3767764Z inflating: build/lib/libtorch_python.so 2024-05-19T18:32:28.3833294Z inflating: build/lib/libtorchbind_test.so 2024-05-19T18:32:28.3853253Z inflating: build/lib/libjitbackend_test.so 2024-05-19T18:32:28.3878004Z inflating: build/lib/libbackend_with_compiler.so 2024-05-19T18:32:28.3911328Z inflating: build/lib/libnnapi_backend.so 2024-05-19T18:32:28.3911817Z creating: build/bin/ 2024-05-19T18:32:28.3960805Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-05-19T18:32:28.4010509Z inflating: build/bin/c10_DeviceGuard_test 2024-05-19T18:32:28.4059466Z inflating: build/bin/c10_Device_test 2024-05-19T18:32:28.4117385Z inflating: build/bin/c10_DispatchKeySet_test 2024-05-19T18:32:28.4169541Z inflating: build/bin/c10_Scalar_test 2024-05-19T18:32:28.4217061Z inflating: build/bin/c10_StreamGuard_test 2024-05-19T18:32:28.4266779Z inflating: build/bin/c10_SymInt_test 2024-05-19T18:32:28.4320019Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-05-19T18:32:28.4374158Z inflating: build/bin/c10_InlineStreamGuard_test 2024-05-19T18:32:28.4428898Z inflating: build/bin/c10_SizesAndStrides_test 2024-05-19T18:32:28.4498362Z inflating: build/bin/c10_cow_test 2024-05-19T18:32:28.4550183Z inflating: build/bin/c10_Bitset_test 2024-05-19T18:32:28.4598400Z inflating: build/bin/c10_ConstexprCrc_test 2024-05-19T18:32:28.4647007Z inflating: build/bin/c10_DeadlockDetection_test 2024-05-19T18:32:28.4696464Z inflating: build/bin/c10_Half_test 2024-05-19T18:32:28.4751348Z inflating: build/bin/c10_LeftRight_test 2024-05-19T18:32:28.4805476Z inflating: build/bin/c10_Metaprogramming_test 2024-05-19T18:32:28.4854117Z inflating: build/bin/c10_Synchronized_test 2024-05-19T18:32:28.4908260Z inflating: build/bin/c10_ThreadLocal_test 2024-05-19T18:32:28.4958546Z inflating: build/bin/c10_TypeIndex_test 2024-05-19T18:32:28.5008219Z inflating: build/bin/c10_TypeList_test 2024-05-19T18:32:28.5055828Z inflating: build/bin/c10_TypeTraits_test 2024-05-19T18:32:28.5106673Z inflating: build/bin/c10_accumulate_test 2024-05-19T18:32:28.5160990Z inflating: build/bin/c10_bfloat16_test 2024-05-19T18:32:28.5209619Z inflating: build/bin/c10_bit_cast_test 2024-05-19T18:32:28.5265112Z inflating: build/bin/c10_complex_math_test 2024-05-19T18:32:28.5318905Z inflating: build/bin/c10_complex_test 2024-05-19T18:32:28.5369949Z inflating: build/bin/c10_exception_test 2024-05-19T18:32:28.5418975Z inflating: build/bin/c10_flags_test 2024-05-19T18:32:28.5468004Z inflating: build/bin/c10_generic_math_test 2024-05-19T18:32:28.5628304Z inflating: build/bin/c10_intrusive_ptr_test 2024-05-19T18:32:28.5677514Z inflating: build/bin/c10_irange_test 2024-05-19T18:32:28.5730121Z inflating: build/bin/c10_lazy_test 2024-05-19T18:32:28.5786359Z inflating: build/bin/c10_logging_test 2024-05-19T18:32:28.5858921Z inflating: build/bin/c10_optional_test 2024-05-19T18:32:28.5919739Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-05-19T18:32:28.5972216Z inflating: build/bin/c10_registry_test 2024-05-19T18:32:28.6118634Z inflating: build/bin/c10_small_vector_test 2024-05-19T18:32:28.6168931Z inflating: build/bin/c10_ssize_test 2024-05-19T18:32:28.6219498Z inflating: build/bin/c10_string_util_test 2024-05-19T18:32:28.6276853Z inflating: build/bin/c10_string_view_test 2024-05-19T18:32:28.6326314Z inflating: build/bin/c10_tempfile_test 2024-05-19T18:32:28.6381443Z inflating: build/bin/c10_typeid_test 2024-05-19T18:32:28.6428322Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-05-19T18:32:28.6865092Z inflating: build/bin/protoc-3.13.0.0 2024-05-19T18:32:28.7301562Z inflating: build/bin/protoc 2024-05-19T18:32:28.7353976Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-05-19T18:32:28.7405173Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-05-19T18:32:28.7456943Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-05-19T18:32:28.7507545Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-05-19T18:32:28.7559345Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-05-19T18:32:28.7610958Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-05-19T18:32:28.7658814Z inflating: build/bin/c10_cuda_CUDATest 2024-05-19T18:32:28.7710921Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-05-19T18:32:28.8031154Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-05-19T18:32:28.8364489Z inflating: build/bin/vec_test_all_types_AVX512 2024-05-19T18:32:28.8714288Z inflating: build/bin/vec_test_all_types_AVX2 2024-05-19T18:32:28.8765808Z inflating: build/bin/FileStoreTest 2024-05-19T18:32:28.8820760Z inflating: build/bin/TCPStoreTest 2024-05-19T18:32:28.8872378Z inflating: build/bin/HashStoreTest 2024-05-19T18:32:28.8885553Z inflating: build/bin/ProcessGroupMPITest 2024-05-19T18:32:28.8938013Z inflating: build/bin/test_edge_op_registration 2024-05-19T18:32:28.8942049Z inflating: build/bin/torch_shm_manager 2024-05-19T18:32:28.8944852Z inflating: build/bin/example_allreduce 2024-05-19T18:32:28.8998810Z inflating: build/bin/test_dist_autograd 2024-05-19T18:32:28.9066494Z inflating: build/bin/test_cpp_rpc 2024-05-19T18:32:28.9068661Z inflating: build/bin/parallel_benchmark 2024-05-19T18:32:28.9134918Z inflating: build/bin/test_mobile_nnc 2024-05-19T18:32:28.9143121Z inflating: build/bin/aot_model_compiler_test 2024-05-19T18:32:28.9479119Z inflating: build/bin/test_lazy 2024-05-19T18:32:29.0633776Z inflating: build/bin/test_api 2024-05-19T18:32:29.0706065Z inflating: build/bin/Dict_test 2024-05-19T18:32:29.0756981Z inflating: build/bin/Dimname_test 2024-05-19T18:32:29.0819978Z inflating: build/bin/MaybeOwned_test 2024-05-19T18:32:29.0875902Z inflating: build/bin/NamedTensor_test 2024-05-19T18:32:29.0934005Z inflating: build/bin/apply_utils_test 2024-05-19T18:32:29.0991105Z inflating: build/bin/atest 2024-05-19T18:32:29.1052589Z inflating: build/bin/basic 2024-05-19T18:32:29.1105844Z inflating: build/bin/broadcast_test 2024-05-19T18:32:29.1155503Z inflating: build/bin/cpu_allocator_test 2024-05-19T18:32:29.1212417Z inflating: build/bin/cpu_generator_test 2024-05-19T18:32:29.1264419Z inflating: build/bin/cpu_profiling_allocator_test 2024-05-19T18:32:29.1355567Z inflating: build/bin/cpu_rng_test 2024-05-19T18:32:29.1404623Z inflating: build/bin/dispatch_key_set_test 2024-05-19T18:32:29.1454290Z inflating: build/bin/dlconvertor_test 2024-05-19T18:32:29.1510576Z inflating: build/bin/extension_backend_test 2024-05-19T18:32:29.1564683Z inflating: build/bin/half_test 2024-05-19T18:32:29.1658385Z inflating: build/bin/ivalue_test 2024-05-19T18:32:29.1706815Z inflating: build/bin/lazy_tensor_test 2024-05-19T18:32:29.1760708Z inflating: build/bin/math_kernel_test 2024-05-19T18:32:29.1815658Z inflating: build/bin/memory_format_test 2024-05-19T18:32:29.1867772Z inflating: build/bin/memory_overlapping_test 2024-05-19T18:32:29.1919398Z inflating: build/bin/mobile_memory_cleanup 2024-05-19T18:32:29.1974053Z inflating: build/bin/native_test 2024-05-19T18:32:29.2024109Z inflating: build/bin/operator_name_test 2024-05-19T18:32:29.2074437Z inflating: build/bin/operators_test 2024-05-19T18:32:29.2125118Z inflating: build/bin/packedtensoraccessor_test 2024-05-19T18:32:29.2190362Z inflating: build/bin/pow_test 2024-05-19T18:32:29.2246527Z inflating: build/bin/quantized_test 2024-05-19T18:32:29.2295062Z inflating: build/bin/reduce_ops_test 2024-05-19T18:32:29.2345446Z inflating: build/bin/reportMemoryUsage_test 2024-05-19T18:32:29.2400790Z inflating: build/bin/scalar_tensor_test 2024-05-19T18:32:29.2458486Z inflating: build/bin/scalar_test 2024-05-19T18:32:29.2508939Z inflating: build/bin/StorageUtils_test 2024-05-19T18:32:29.2560376Z inflating: build/bin/stride_properties_test 2024-05-19T18:32:29.2637096Z inflating: build/bin/tensor_iterator_test 2024-05-19T18:32:29.2690527Z inflating: build/bin/test_parallel 2024-05-19T18:32:29.2693237Z inflating: build/bin/thread_init_test 2024-05-19T18:32:29.2748229Z inflating: build/bin/type_ptr_test 2024-05-19T18:32:29.2806786Z inflating: build/bin/type_test 2024-05-19T18:32:29.2858277Z inflating: build/bin/undefined_tensor_test 2024-05-19T18:32:29.2859586Z inflating: build/bin/verify_api_visibility 2024-05-19T18:32:29.2926614Z inflating: build/bin/legacy_vmap_test 2024-05-19T18:32:29.2976977Z inflating: build/bin/weakref_test 2024-05-19T18:32:29.3026959Z inflating: build/bin/wrapdim_test 2024-05-19T18:32:29.3077604Z inflating: build/bin/xla_tensor_test 2024-05-19T18:32:29.3136022Z inflating: build/bin/IListRef_test 2024-05-19T18:32:29.3239542Z inflating: build/bin/List_test 2024-05-19T18:32:29.3304832Z inflating: build/bin/KernelFunction_test 2024-05-19T18:32:29.3423978Z inflating: build/bin/kernel_function_legacy_test 2024-05-19T18:32:29.3518756Z inflating: build/bin/kernel_function_test 2024-05-19T18:32:29.3644156Z inflating: build/bin/kernel_lambda_legacy_test 2024-05-19T18:32:29.3744759Z inflating: build/bin/kernel_lambda_test 2024-05-19T18:32:29.3804565Z inflating: build/bin/kernel_stackbased_test 2024-05-19T18:32:29.3898705Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-05-19T18:32:29.3948138Z inflating: build/bin/CppSignature_test 2024-05-19T18:32:29.3995972Z inflating: build/bin/op_allowlist_test 2024-05-19T18:32:29.4050589Z inflating: build/bin/backend_fallback_test 2024-05-19T18:32:29.4344642Z inflating: build/bin/op_registration_test 2024-05-19T18:32:29.4407056Z inflating: build/bin/inline_container_test 2024-05-19T18:32:29.4458879Z inflating: build/bin/cuda_apply_test 2024-05-19T18:32:29.4508938Z inflating: build/bin/cuda_allocator_test 2024-05-19T18:32:29.4561931Z inflating: build/bin/cuda_caching_host_allocator_test 2024-05-19T18:32:29.4619737Z inflating: build/bin/cuda_atomic_ops_test 2024-05-19T18:32:29.4687522Z inflating: build/bin/cuda_complex_math_test 2024-05-19T18:32:29.4745688Z inflating: build/bin/cuda_complex_test 2024-05-19T18:32:29.4794002Z inflating: build/bin/cuda_device_test 2024-05-19T18:32:29.4851099Z inflating: build/bin/cuda_cub_test 2024-05-19T18:32:29.4901203Z inflating: build/bin/cuda_dlconvertor_test 2024-05-19T18:32:29.4964133Z inflating: build/bin/cuda_distributions_test 2024-05-19T18:32:29.5019748Z inflating: build/bin/cuda_generator_test 2024-05-19T18:32:29.5067752Z inflating: build/bin/cuda_half_test 2024-05-19T18:32:29.5117884Z inflating: build/bin/cuda_integer_divider_test 2024-05-19T18:32:29.5166199Z inflating: build/bin/cuda_optional_test 2024-05-19T18:32:29.5216766Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-05-19T18:32:29.5267946Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-05-19T18:32:29.5316773Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-05-19T18:32:29.5376317Z inflating: build/bin/cuda_stream_test 2024-05-19T18:32:29.5424021Z inflating: build/bin/cuda_cudnn_test 2024-05-19T18:32:29.5475000Z inflating: build/bin/cuda_vectorized_test 2024-05-19T18:32:29.5489454Z inflating: build/bin/tutorial_tensorexpr 2024-05-19T18:32:29.5553912Z inflating: build/bin/ProcessGroupGlooTest 2024-05-19T18:32:29.5609477Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-05-19T18:32:29.5670524Z inflating: build/bin/ProcessGroupNCCLTest 2024-05-19T18:32:29.5730817Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-05-19T18:32:29.6541085Z inflating: build/bin/test_tensorexpr 2024-05-19T18:32:29.7104082Z inflating: build/bin/test_jit 2024-05-19T18:32:29.7104584Z creating: .additional_ci_files/ 2024-05-19T18:32:29.7144601Z inflating: .additional_ci_files/test-times.json 2024-05-19T18:32:29.7306920Z inflating: .additional_ci_files/test-class-times.json 2024-05-19T18:32:29.7336317Z ##[group]Run rm artifacts.zip 2024-05-19T18:32:29.7336718Z rm artifacts.zip 2024-05-19T18:32:29.7344353Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:29.7344866Z env: 2024-05-19T18:32:29.7345166Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:29.7345628Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:29.7346120Z ##[endgroup] 2024-05-19T18:32:29.8047827Z ##[group]Run df -H 2024-05-19T18:32:29.8048288Z df -H 2024-05-19T18:32:29.8058143Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:29.8058876Z env: 2024-05-19T18:32:29.8059257Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:29.8059902Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:29.8060436Z ##[endgroup] 2024-05-19T18:32:29.8089735Z Filesystem Size Used Avail Use% Mounted on 2024-05-19T18:32:29.8090282Z devtmpfs 34G 0 34G 0% /dev 2024-05-19T18:32:29.8090769Z tmpfs 34G 0 34G 0% /dev/shm 2024-05-19T18:32:29.8091423Z tmpfs 34G 455k 34G 1% /run 2024-05-19T18:32:29.8092232Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-05-19T18:32:29.8092868Z /dev/nvme0n1p1 162G 44G 118G 27% / 2024-05-19T18:32:29.8132088Z Prepare all required actions 2024-05-19T18:32:29.8132525Z Getting action download info 2024-05-19T18:32:29.9231654Z ##[group]Run ./.github/actions/download-td-artifacts 2024-05-19T18:32:29.9232119Z with: 2024-05-19T18:32:29.9232386Z env: 2024-05-19T18:32:29.9232703Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:29.9233400Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:29.9233990Z ##[endgroup] 2024-05-19T18:32:29.9292991Z ##[group]Run seemethere/download-artifact-s3@v4 2024-05-19T18:32:29.9293440Z with: 2024-05-19T18:32:29.9293717Z name: td_results 2024-05-19T18:32:29.9294036Z s3-bucket: gha-artifacts 2024-05-19T18:32:29.9294386Z region: us-east-1 2024-05-19T18:32:29.9294691Z env: 2024-05-19T18:32:29.9294962Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:29.9295419Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:29.9295901Z ##[endgroup] 2024-05-19T18:32:30.3405809Z (node:28867) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-05-19T18:32:30.3406501Z 2024-05-19T18:32:30.3406766Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-05-19T18:32:30.3407500Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-05-19T18:32:30.3408332Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-05-19T18:32:30.4182983Z Found 0 objects with prefix pytorch/pytorch/9148947323/td_results/ 2024-05-19T18:32:30.4189718Z Artifact download has finished successfully 2024-05-19T18:32:30.4326573Z ##[group]Run mkdir -p .additional_ci_files 2024-05-19T18:32:30.4327068Z mkdir -p .additional_ci_files 2024-05-19T18:32:30.4327634Z mv td_results.json .additional_ci_files/td_results.json 2024-05-19T18:32:30.4335691Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:30.4336191Z env: 2024-05-19T18:32:30.4336482Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:30.4336958Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:30.4337449Z ##[endgroup] 2024-05-19T18:32:30.4370824Z mv: cannot stat 'td_results.json': No such file or directory 2024-05-19T18:32:30.4376017Z ##[error]Process completed with exit code 1. 2024-05-19T18:32:30.4409013Z ##[group]Run .github/scripts/parse_ref.py 2024-05-19T18:32:30.4409495Z .github/scripts/parse_ref.py 2024-05-19T18:32:30.4417029Z shell: /usr/bin/bash -e {0} 2024-05-19T18:32:30.4417404Z env: 2024-05-19T18:32:30.4417693Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:30.4418151Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:30.4418630Z ##[endgroup] 2024-05-19T18:32:30.4647940Z Prepare all required actions 2024-05-19T18:32:30.4685923Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-05-19T18:32:30.4686385Z with: 2024-05-19T18:32:30.4686953Z github-token: *** 2024-05-19T18:32:30.4687261Z env: 2024-05-19T18:32:30.4687544Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:30.4688010Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:30.4688498Z ##[endgroup] 2024-05-19T18:32:30.4704269Z ##[group]Run set -eux 2024-05-19T18:32:30.4704614Z set -eux 2024-05-19T18:32:30.4705212Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-05-19T18:32:30.4713025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:30.4713530Z env: 2024-05-19T18:32:30.4713819Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:30.4714294Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:30.4714939Z GITHUB_TOKEN: *** 2024-05-19T18:32:30.4715252Z ##[endgroup] 2024-05-19T18:32:30.4735260Z + python3 .github/scripts/get_workflow_job_id.py 9148947323 i-0f3ccc189396031c1 2024-05-19T18:32:32.0168077Z setting job-id=25153197363 2024-05-19T18:32:32.0169763Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:32:32.0364894Z Prepare all required actions 2024-05-19T18:32:32.0365349Z Getting action download info 2024-05-19T18:32:32.1449570Z ##[group]Run ./.github/actions/filter-test-configs 2024-05-19T18:32:32.1450043Z with: 2024-05-19T18:32:32.1450505Z github-token: *** 2024-05-19T18:32:32.1458348Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-05-19T18:32:32.1467004Z job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:32:32.1467728Z env: 2024-05-19T18:32:32.1468010Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:32.1468474Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:32.1468962Z ##[endgroup] 2024-05-19T18:32:32.1510402Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T18:32:32.1511065Z with: 2024-05-19T18:32:32.1511344Z shell: bash 2024-05-19T18:32:32.1511648Z timeout_minutes: 10 2024-05-19T18:32:32.1511974Z max_attempts: 5 2024-05-19T18:32:32.1512298Z retry_wait_seconds: 30 2024-05-19T18:32:32.1513084Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:32:32.1513919Z polling_interval_seconds: 1 2024-05-19T18:32:32.1514287Z warning_on_retry: true 2024-05-19T18:32:32.1514640Z continue_on_error: false 2024-05-19T18:32:32.1514975Z env: 2024-05-19T18:32:32.1515256Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:32.1515700Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:32.1516347Z GITHUB_TOKEN: *** 2024-05-19T18:32:32.1516662Z ##[endgroup] 2024-05-19T18:32:32.2025192Z + python3 -m pip install requests==2.26.0 pyyaml==6.0.1 2024-05-19T18:32:32.4172968Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T18:32:32.5407552Z Collecting requests==2.26.0 2024-05-19T18:32:32.5608715Z Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB) 2024-05-19T18:32:32.7173658Z Collecting pyyaml==6.0.1 2024-05-19T18:32:32.7237594Z Downloading PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (670 kB) 2024-05-19T18:32:32.8070201Z Collecting certifi>=2017.4.17 2024-05-19T18:32:32.8119046Z Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) 2024-05-19T18:32:32.8552002Z Collecting idna<4,>=2.5; python_version >= "3" 2024-05-19T18:32:32.8594692Z Downloading idna-3.7-py3-none-any.whl (66 kB) 2024-05-19T18:32:32.9741550Z Collecting urllib3<1.27,>=1.21.1 2024-05-19T18:32:32.9802013Z Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-05-19T18:32:33.3207208Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2024-05-19T18:32:33.3253278Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-05-19T18:32:33.3993543Z Installing collected packages: certifi, idna, urllib3, charset-normalizer, requests, pyyaml 2024-05-19T18:32:33.5624991Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2024-05-19T18:32:33.5626214Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-05-19T18:32:33.6523185Z Successfully installed certifi-2024.2.2 charset-normalizer-2.0.12 idna-3.7 pyyaml-6.0.1 requests-2.26.0 urllib3-1.26.18 2024-05-19T18:32:34.2021037Z Command completed after 1 attempt(s). 2024-05-19T18:32:34.2064049Z ##[group]Run set -x 2024-05-19T18:32:34.2064397Z set -x 2024-05-19T18:32:34.2064703Z  2024-05-19T18:32:34.2065261Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:32:34.2065942Z # in runner workspace 2024-05-19T18:32:34.2066474Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-05-19T18:32:34.2074368Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:34.2074876Z env: 2024-05-19T18:32:34.2075169Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:34.2075627Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:34.2076122Z ##[endgroup] 2024-05-19T18:32:34.2096977Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-05-19T18:32:34.2293364Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:32:34.2293915Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-05-19T18:32:34.2294381Z echo "Job name: ${JOB_NAME}" 2024-05-19T18:32:34.2294786Z  2024-05-19T18:32:34.2295335Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-05-19T18:32:34.2296016Z # in runner workspace 2024-05-19T18:32:34.2296589Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-05-19T18:32:34.2297245Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-05-19T18:32:34.2297718Z  --job-name "${JOB_NAME}" \ 2024-05-19T18:32:34.2305653Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-05-19T18:32:34.2313943Z  --selected-test-configs "" \ 2024-05-19T18:32:34.2314410Z  --pr-number "${PR_NUMBER}" \ 2024-05-19T18:32:34.2314836Z  --tag "${TAG}" \ 2024-05-19T18:32:34.2315228Z  --event-name "${EVENT_NAME}" \ 2024-05-19T18:32:34.2315682Z  --schedule "${SCHEDULE}" \ 2024-05-19T18:32:34.2316124Z  --branch "${HEAD_BRANCH}" 2024-05-19T18:32:34.2323580Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:34.2324081Z env: 2024-05-19T18:32:34.2324375Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:34.2324839Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:34.2325523Z GITHUB_TOKEN: *** 2024-05-19T18:32:34.2326178Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:32:34.2326915Z PR_NUMBER: 2024-05-19T18:32:34.2327239Z TAG: ciflow/inductor/121956 2024-05-19T18:32:34.2327618Z EVENT_NAME: push 2024-05-19T18:32:34.2327933Z SCHEDULE: 2024-05-19T18:32:34.2328232Z HEAD_BRANCH: 2024-05-19T18:32:34.2328532Z ##[endgroup] 2024-05-19T18:32:34.2347129Z Workflow: inductor 2024-05-19T18:32:34.2348014Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:32:34.5376311Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-05-19T18:32:34.6458771Z ##[group]Run echo "Filtered matrix:" 2024-05-19T18:32:34.6459231Z echo "Filtered matrix:" 2024-05-19T18:32:34.6467194Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-05-19T18:32:34.6475447Z  2024-05-19T18:32:34.6475727Z echo 2024-05-19T18:32:34.6476114Z echo "Is the current job unstable? False" 2024-05-19T18:32:34.6476579Z  2024-05-19T18:32:34.6476853Z echo 2024-05-19T18:32:34.6477218Z echo "Is keep-going label set? False" 2024-05-19T18:32:34.6477989Z  2024-05-19T18:32:34.6478283Z echo 2024-05-19T18:32:34.6478724Z echo "Renabled issues? " 2024-05-19T18:32:34.6486527Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:34.6487037Z env: 2024-05-19T18:32:34.6487331Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:34.6487791Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:34.6488283Z ##[endgroup] 2024-05-19T18:32:34.6508005Z Filtered matrix: 2024-05-19T18:32:34.6521662Z {include: [{config: inductor, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-05-19T18:32:34.6531062Z 2024-05-19T18:32:34.6531220Z Is the current job unstable? False 2024-05-19T18:32:34.6531538Z 2024-05-19T18:32:34.6531770Z Is keep-going label set? False 2024-05-19T18:32:34.6532038Z 2024-05-19T18:32:34.6532163Z Renabled issues? 2024-05-19T18:32:34.6573172Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:32:34.6573926Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-05-19T18:32:34.6581617Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T18:32:34.6582112Z env: 2024-05-19T18:32:34.6582404Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:34.6582863Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:34.6583353Z JOB_TIMEOUT: 240 2024-05-19T18:32:34.6583662Z ##[endgroup] 2024-05-19T18:32:34.6671984Z ##[group]Run set -x 2024-05-19T18:32:34.6672397Z set -x 2024-05-19T18:32:34.6672714Z  2024-05-19T18:32:34.6673072Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-05-19T18:32:34.6673616Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-05-19T18:32:34.6674182Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-05-19T18:32:34.6674696Z  TEST_COMMAND=.ci/onnx/test.sh 2024-05-19T18:32:34.6675110Z else 2024-05-19T18:32:34.6675452Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:32:34.6675876Z fi 2024-05-19T18:32:34.6676158Z  2024-05-19T18:32:34.6676624Z # detached container should get cleaned up by teardown_ec2_linux 2024-05-19T18:32:34.6677372Z # TODO: Stop building test binaries as part of the build phase 2024-05-19T18:32:34.6678502Z # Used for GPU_FLAG since that doesn't play nice 2024-05-19T18:32:34.6679201Z # shellcheck disable=SC2086,SC2090 2024-05-19T18:32:34.6679691Z container_name=$(docker run \ 2024-05-19T18:32:34.6680237Z  ${GPU_FLAG:-} \ 2024-05-19T18:32:34.6680622Z  -e BUILD_ENVIRONMENT \ 2024-05-19T18:32:34.6681024Z  -e PR_NUMBER \ 2024-05-19T18:32:34.6681387Z  -e GITHUB_ACTIONS \ 2024-05-19T18:32:34.6681782Z  -e GITHUB_REPOSITORY \ 2024-05-19T18:32:34.6682194Z  -e GITHUB_WORKFLOW \ 2024-05-19T18:32:34.6682588Z  -e GITHUB_JOB \ 2024-05-19T18:32:34.6682950Z  -e GITHUB_RUN_ID \ 2024-05-19T18:32:34.6683342Z  -e GITHUB_RUN_NUMBER \ 2024-05-19T18:32:34.6683753Z  -e GITHUB_RUN_ATTEMPT \ 2024-05-19T18:32:34.6684147Z  -e JOB_ID \ 2024-05-19T18:32:34.6684493Z  -e JOB_NAME \ 2024-05-19T18:32:34.6684850Z  -e BASE_SHA \ 2024-05-19T18:32:34.6685201Z  -e BRANCH \ 2024-05-19T18:32:34.6685535Z  -e SHA1 \ 2024-05-19T18:32:34.6685890Z  -e AWS_DEFAULT_REGION \ 2024-05-19T18:32:34.6686300Z  -e IN_WHEEL_TEST \ 2024-05-19T18:32:34.6686693Z  -e SHARD_NUMBER \ 2024-05-19T18:32:34.6687065Z  -e TEST_CONFIG \ 2024-05-19T18:32:34.6687444Z  -e NUM_TEST_SHARDS \ 2024-05-19T18:32:34.6687841Z  -e REENABLED_ISSUES \ 2024-05-19T18:32:34.6688260Z  -e CONTINUE_THROUGH_ERROR \ 2024-05-19T18:32:34.6688686Z  -e VERBOSE_TEST_LOGS \ 2024-05-19T18:32:34.6689096Z  -e NO_TEST_TIMEOUT \ 2024-05-19T18:32:34.6689477Z  -e NO_TD \ 2024-05-19T18:32:34.6689830Z  -e TD_DISTRIBUTED \ 2024-05-19T18:32:34.6690209Z  -e PR_LABELS \ 2024-05-19T18:32:34.6690622Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-05-19T18:32:34.6691088Z  -e SCCACHE_BUCKET \ 2024-05-19T18:32:34.6691483Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-05-19T18:32:34.6691900Z  -e XLA_CUDA \ 2024-05-19T18:32:34.6692311Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-05-19T18:32:34.6692830Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-05-19T18:32:34.6693359Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-05-19T18:32:34.6693880Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-05-19T18:32:34.6694354Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-05-19T18:32:34.6694782Z  -e DASHBOARD_TAG \ 2024-05-19T18:32:34.6695260Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-05-19T18:32:34.6695819Z  --security-opt seccomp=unconfined \ 2024-05-19T18:32:34.6696291Z  --cap-add=SYS_PTRACE \ 2024-05-19T18:32:34.6696691Z  --ipc=host \ 2024-05-19T18:32:34.6697047Z  --shm-size="${SHM_SIZE}" \ 2024-05-19T18:32:34.6697451Z  --tty \ 2024-05-19T18:32:34.6697775Z  --detach \ 2024-05-19T18:32:34.6698144Z  --name="${container_name}" \ 2024-05-19T18:32:34.6698561Z  --user jenkins \ 2024-05-19T18:32:34.6699136Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-05-19T18:32:34.6699707Z  -w /var/lib/jenkins/workspace \ 2024-05-19T18:32:34.6700153Z  "${DOCKER_IMAGE}" 2024-05-19T18:32:34.6700506Z ) 2024-05-19T18:32:34.6700918Z # Propagate download.pytorch.org IP to container 2024-05-19T18:32:34.6701838Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-05-19T18:32:34.6702805Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-05-19T18:32:34.6703715Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-05-19T18:32:34.6711116Z shell: /usr/bin/bash -e {0} 2024-05-19T18:32:34.6711520Z env: 2024-05-19T18:32:34.6711805Z GIT_DEFAULT_BRANCH: main 2024-05-19T18:32:34.6712264Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:34.6712876Z BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:32:34.6713406Z PR_NUMBER: 2024-05-19T18:32:34.6713775Z GITHUB_REPOSITORY: pytorch/pytorch 2024-05-19T18:32:34.6714260Z GITHUB_WORKFLOW: inductor 2024-05-19T18:32:34.6714615Z GITHUB_JOB: test 2024-05-19T18:32:34.6714938Z GITHUB_RUN_ID: 9148947323 2024-05-19T18:32:34.6715304Z GITHUB_RUN_NUMBER: 69546 2024-05-19T18:32:34.6715658Z GITHUB_RUN_ATTEMPT: 1 2024-05-19T18:32:34.6716002Z JOB_ID: 25153197363 2024-05-19T18:32:34.6716657Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:32:34.6717378Z BRANCH: 2024-05-19T18:32:34.6717995Z SHA1: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:34.6718553Z BASE_SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:34.6719048Z TEST_CONFIG: inductor_torchbench 2024-05-19T18:32:34.6719445Z SHARD_NUMBER: 1 2024-05-19T18:32:34.6719754Z NUM_TEST_SHARDS: 2 2024-05-19T18:32:34.6720086Z REENABLED_ISSUES: 2024-05-19T18:32:34.6720439Z CONTINUE_THROUGH_ERROR: False 2024-05-19T18:32:34.6720837Z VERBOSE_TEST_LOGS: False 2024-05-19T18:32:34.6721202Z NO_TEST_TIMEOUT: False 2024-05-19T18:32:34.6721541Z NO_TD: False 2024-05-19T18:32:34.6721854Z TD_DISTRIBUTED: False 2024-05-19T18:32:34.6722282Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-05-19T18:32:34.6722772Z SCCACHE_S3_KEY_PREFIX: inductor 2024-05-19T18:32:34.6723162Z SHM_SIZE: 2g 2024-05-19T18:32:34.6724308Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:32:34.6725477Z XLA_CUDA: 2024-05-19T18:32:34.6725961Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:32:34.6726577Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-05-19T18:32:34.6727020Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-05-19T18:32:34.6727436Z DASHBOARD_TAG: 2024-05-19T18:32:34.6727945Z HUGGING_FACE_HUB_TOKEN: *** 2024-05-19T18:32:34.6728316Z ##[endgroup] 2024-05-19T18:32:34.6747332Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-05-19T18:32:34.6748039Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]] 2024-05-19T18:32:34.6748556Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-05-19T18:32:34.6754584Z +++ nproc --ignore=2 2024-05-19T18:32:34.6774657Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9148947323 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T18:32:46.0807779Z + container_name=71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T18:32:46.0808723Z + grep download.pytorch.org /etc/hosts 2024-05-19T18:32:46.0810415Z + docker exec -i 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 sudo bash -c '/bin/cat >> /etc/hosts' 2024-05-19T18:32:46.1482461Z + echo DOCKER_CONTAINER_ID=71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T18:32:46.1485353Z ++ echo dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl 2024-05-19T18:32:46.1487261Z + docker exec -t 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 sh -c 'pip install dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-05-19T18:32:46.5276667Z Processing ./dist/torch-2.4.0a0+gitc8c7ddf-cp310-cp310-linux_x86_64.whl (from torch==2.4.0a0+gitc8c7ddf) 2024-05-19T18:32:46.8426989Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.13.1) 2024-05-19T18:32:46.8429532Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (4.11.0) 2024-05-19T18:32:46.8433528Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.12) 2024-05-19T18:32:46.8436800Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.8.8) 2024-05-19T18:32:46.8440454Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.1.4) 2024-05-19T18:32:46.8443843Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2024.2.0) 2024-05-19T18:32:46.8470011Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (3.3.0) 2024-05-19T18:32:46.8529268Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.21.2) 2024-05-19T18:32:46.8951094Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (2.1.5) 2024-05-19T18:32:46.9100973Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.4.0a0+gitc8c7ddf->torch==2.4.0a0+gitc8c7ddf) (1.3.0) 2024-05-19T18:32:48.2171947Z Installing collected packages: torch 2024-05-19T18:32:56.8744344Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-05-19T18:32:56.8745875Z timm 0.9.7 requires torchvision, which is not installed. 2024-05-19T18:32:56.8746766Z Successfully installed torch-2.4.0a0+gitc8c7ddf 2024-05-19T18:32:56.9542870Z ++ dirname .ci/pytorch/test.sh 2024-05-19T18:32:56.9546037Z + source .ci/pytorch/common.sh 2024-05-19T18:32:56.9548718Z +++ dirname .ci/pytorch/common.sh 2024-05-19T18:32:56.9555836Z ++ source .ci/pytorch/common_utils.sh 2024-05-19T18:32:56.9558229Z +++ declare -f -t trap_add 2024-05-19T18:32:56.9562717Z ++ set -ex 2024-05-19T18:32:56.9563375Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:32:56.9563977Z ++ BUILD_TEST_LIBTORCH=0 2024-05-19T18:32:56.9564677Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-05-19T18:32:56.9566552Z ++ stat -c %u /var/lib/jenkins/workspace 2024-05-19T18:32:56.9574108Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-05-19T18:32:56.9574654Z + trap_add cleanup_workspace EXIT 2024-05-19T18:32:56.9575218Z + trap_add_cmd=cleanup_workspace 2024-05-19T18:32:56.9575632Z + shift 2024-05-19T18:32:56.9575934Z + for trap_add_name in "$@" 2024-05-19T18:32:56.9579626Z +++ trap -p EXIT 2024-05-19T18:32:56.9580957Z ++ eval 'extract_trap_cmd ' 2024-05-19T18:32:56.9581335Z +++ extract_trap_cmd 2024-05-19T18:32:56.9581725Z +++ printf '%s\n' '' 2024-05-19T18:32:56.9582195Z ++ printf '%s\n' cleanup_workspace 2024-05-19T18:32:56.9582911Z + trap -- ' 2024-05-19T18:32:56.9583253Z cleanup_workspace' EXIT 2024-05-19T18:32:56.9583724Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-05-19T18:32:57.3700731Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-05-19T18:32:57.3713868Z + echo 'Environment variables:' 2024-05-19T18:32:57.3714294Z Environment variables: 2024-05-19T18:32:57.3714620Z + env 2024-05-19T18:32:57.3719819Z INSTALLED_DB=yes 2024-05-19T18:32:57.3720202Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:32:57.3720619Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:32:57.3721125Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:32:57.3721946Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:32:57.3722742Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:32:57.3723437Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:32:57.3724196Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:32:57.3725028Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:32:57.3725800Z HOSTNAME=71bd051df825 2024-05-19T18:32:57.3726970Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.3728034Z GITHUB_ACTION=__self 2024-05-19T18:32:57.3728395Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:32:57.3734525Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:32:57.3740347Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:32:57.3740859Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:32:57.3741213Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:32:57.3741568Z TEST_CONFIG=inductor_torchbench 2024-05-19T18:32:57.3755180Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:32:57.3755719Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:32:57.3756187Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:32:57.3756633Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:32:57.3757056Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:32:57.3757500Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:32:57.3758461Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:32:57.3758937Z GITHUB_REF_TYPE=tag 2024-05-19T18:32:57.3759274Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:32:57.3759662Z NCCL_VERSION=2.20.5-1 2024-05-19T18:32:57.3760065Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.3760515Z XLA_CUDA= 2024-05-19T18:32:57.3761026Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:32:57.3761694Z *** 2024-05-19T18:32:57.3761986Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:32:57.3762390Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:32:57.3762753Z GITHUB_ACTIONS=true 2024-05-19T18:32:57.3763092Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:57.3763583Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:32:57.3764105Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:32:57.3764587Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.3765098Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:32:57.3765597Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.3766385Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:32:57.3767257Z UCC_HOME=/usr 2024-05-19T18:32:57.3767601Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:32:57.3768131Z VERBOSE_TEST_LOGS=False 2024-05-19T18:32:57.3768602Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:32:57.3769215Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:32:57.3769904Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:32:57.3770528Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:32:57.3771027Z SHARD_NUMBER=1 2024-05-19T18:32:57.3771459Z GITHUB_REF_PROTECTED=false 2024-05-19T18:32:57.3771951Z HOME=/var/lib/jenkins 2024-05-19T18:32:57.3772453Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:32:57.3773080Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:32:57.3773755Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:32:57.3774422Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:32:57.3774929Z CUDA_VERSION=12.4.0 2024-05-19T18:32:57.3775534Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:32:57.3776139Z NUM_TEST_SHARDS=2 2024-05-19T18:32:57.3776558Z UCX_HOME=/usr 2024-05-19T18:32:57.3777264Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:32:57.3778807Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.3780688Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:32:57.3782610Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.3784366Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:32:57.3785412Z GITHUB_EVENT_NAME=push 2024-05-19T18:32:57.3785914Z DASHBOARD_TAG= 2024-05-19T18:32:57.3786357Z GITHUB_RUN_ID=9148947323 2024-05-19T18:32:57.3787027Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:32:57.3787805Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:32:57.3789344Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.3790822Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:32:57.3791586Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:32:57.3792112Z PR_NUMBER= 2024-05-19T18:32:57.3792530Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:32:57.3793037Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:32:57.3793710Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:32:57.3794446Z TERM=xterm 2024-05-19T18:32:57.3794942Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:32:57.3795500Z INSTALLED_VISION=yes 2024-05-19T18:32:57.3795945Z BRANCH= 2024-05-19T18:32:57.3796383Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:32:57.3796975Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:32:57.3797888Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:32:57.3799254Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:32:57.3800448Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:32:57.3801185Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:32:57.3801903Z REENABLED_ISSUES= 2024-05-19T18:32:57.3802359Z SHLVL=1 2024-05-19T18:32:57.3802761Z MAX_JOBS=14 2024-05-19T18:32:57.3803261Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:32:57.3803805Z NVARCH=x86_64 2024-05-19T18:32:57.3804251Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:32:57.3804908Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.3805719Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:32:57.3806495Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:32:57.3807460Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:32:57.3808317Z GITHUB_JOB=test 2024-05-19T18:32:57.3808910Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:32:57.3809763Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:32:57.3810750Z NO_TEST_TIMEOUT=False 2024-05-19T18:32:57.3811252Z TD_DISTRIBUTED=False 2024-05-19T18:32:57.3811826Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:32:57.3812489Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:32:57.3813148Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:32:57.3813693Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:32:57.3814179Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:32:57.3814636Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:32:57.3816224Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:32:57.3817901Z GITHUB_BASE_REF= 2024-05-19T18:32:57.3818398Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:32:57.3818961Z CI=true 2024-05-19T18:32:57.3819490Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:32:57.3820137Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:32:57.3820697Z JOB_ID=25153197363 2024-05-19T18:32:57.3821196Z INSTALLED_PROTOBUF=yes 2024-05-19T18:32:57.3821689Z GITHUB_HEAD_REF= 2024-05-19T18:32:57.3822146Z GITHUB_ACTION_REF= 2024-05-19T18:32:57.3822801Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:32:57.3823547Z GITHUB_WORKFLOW=inductor 2024-05-19T18:32:57.3824038Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:32:57.3825458Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.3826746Z NO_TD=False 2024-05-19T18:32:57.3827202Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:32:57.3827758Z _=/usr/bin/env 2024-05-19T18:32:57.3828513Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-05-19T18:32:57.3964656Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-05-19T18:32:57.3965791Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-05-19T18:32:57.3966670Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-05-19T18:32:57.3967550Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-05-19T18:32:57.3968175Z + BUILD_DIR=build 2024-05-19T18:32:57.3968514Z + BUILD_RENAMED_DIR=build_renamed 2024-05-19T18:32:57.3968907Z + BUILD_BIN_DIR=build/bin 2024-05-19T18:32:57.3969259Z + SHARD_NUMBER=1 2024-05-19T18:32:57.3969613Z + NUM_TEST_SHARDS=2 2024-05-19T18:32:57.3970027Z + export VALGRIND=ON 2024-05-19T18:32:57.3970399Z + VALGRIND=ON 2024-05-19T18:32:57.3970874Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]] 2024-05-19T18:32:57.3971379Z + [[ 0 == \1 ]] 2024-05-19T18:32:57.3971679Z + [[ False == \1 ]] 2024-05-19T18:32:57.3972164Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]] 2024-05-19T18:32:57.3972699Z ++ realpath build/custom_test_artifacts 2024-05-19T18:32:57.3976003Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-05-19T18:32:57.3976916Z + [[ -n '' ]] 2024-05-19T18:32:57.3977583Z + echo 'Environment variables' 2024-05-19T18:32:57.3977980Z Environment variables 2024-05-19T18:32:57.3978314Z + env 2024-05-19T18:32:57.3981367Z INSTALLED_DB=yes 2024-05-19T18:32:57.3981873Z NV_LIBCUBLAS_VERSION=12.4.2.65-1 2024-05-19T18:32:57.3982412Z NVIDIA_VISIBLE_DEVICES=all 2024-05-19T18:32:57.3982863Z NV_NVML_DEV_VERSION=12.4.99-1 2024-05-19T18:32:57.3983627Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T18:32:57.3984428Z CONTINUE_THROUGH_ERROR=False 2024-05-19T18:32:57.3985059Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.20.5-1+cuda12.4 2024-05-19T18:32:57.3985752Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:32:57.3986525Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T18:32:57.3987127Z HOSTNAME=71bd051df825 2024-05-19T18:32:57.3988177Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.3989056Z GITHUB_ACTION=__self 2024-05-19T18:32:57.3989441Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-05-19T18:32:57.3995548Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-05-19T18:32:57.4001750Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.2.65-1 2024-05-19T18:32:57.4002273Z NV_NVTX_VERSION=12.4.99-1 2024-05-19T18:32:57.4002644Z GITHUB_RUN_NUMBER=69546 2024-05-19T18:32:57.4003005Z TEST_CONFIG=inductor_torchbench 2024-05-19T18:32:57.4003406Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-05-19T18:32:57.4003883Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-05-19T18:32:57.4004353Z NV_CUDA_CUDART_DEV_VERSION=12.4.99-1 2024-05-19T18:32:57.4004806Z NV_LIBCUSPARSE_VERSION=12.3.0.142-1 2024-05-19T18:32:57.4005233Z NV_LIBNPP_VERSION=12.2.5.2-1 2024-05-19T18:32:57.4005679Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-05-19T18:32:57.4006184Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-05-19T18:32:57.4006657Z GITHUB_REF_TYPE=tag 2024-05-19T18:32:57.4007004Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-05-19T18:32:57.4007392Z NCCL_VERSION=2.20.5-1 2024-05-19T18:32:57.4007800Z BASE_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.4008264Z XLA_CUDA= 2024-05-19T18:32:57.4008792Z HUGGING_FACE_HUB_TOKEN=*** 2024-05-19T18:32:57.4009211Z *** 2024-05-19T18:32:57.4009503Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-05-19T18:32:57.4009905Z GITHUB_REPOSITORY_ID=65600975 2024-05-19T18:32:57.4010336Z GITHUB_ACTIONS=true 2024-05-19T18:32:57.4010681Z NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T18:32:57.4011175Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.99-1 2024-05-19T18:32:57.4011720Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.2-1 2024-05-19T18:32:57.4012204Z SHA1=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.4012730Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-05-19T18:32:57.4013238Z GITHUB_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.4014040Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/121956 2024-05-19T18:32:57.4014749Z UCC_HOME=/usr 2024-05-19T18:32:57.4015214Z NV_LIBCUBLAS_DEV_VERSION=12.4.2.65-1 2024-05-19T18:32:57.4015632Z VERBOSE_TEST_LOGS=False 2024-05-19T18:32:57.4015991Z NVIDIA_PRODUCT_NAME=CUDA 2024-05-19T18:32:57.4016452Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-05-19T18:32:57.4016959Z GITHUB_REF=refs/tags/ciflow/inductor/121956 2024-05-19T18:32:57.4017429Z NV_CUDA_CUDART_VERSION=12.4.99-1 2024-05-19T18:32:57.4017805Z SHARD_NUMBER=1 2024-05-19T18:32:57.4018125Z GITHUB_REF_PROTECTED=false 2024-05-19T18:32:57.4018486Z HOME=/var/lib/jenkins 2024-05-19T18:32:57.4018857Z GITHUB_API_URL=https://api.github.com 2024-05-19T18:32:57.4019300Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-05-19T18:32:57.4019788Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-05-19T18:32:57.4020282Z SCCACHE_S3_KEY_PREFIX=inductor 2024-05-19T18:32:57.4020667Z CUDA_VERSION=12.4.0 2024-05-19T18:32:57.4021086Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.2.65-1 2024-05-19T18:32:57.4021536Z NUM_TEST_SHARDS=2 2024-05-19T18:32:57.4021851Z UCX_HOME=/usr 2024-05-19T18:32:57.4022371Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.0-1 2024-05-19T18:32:57.4023554Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.4024768Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-05-19T18:32:57.4025972Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.4027085Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-05-19T18:32:57.4027737Z GITHUB_EVENT_NAME=push 2024-05-19T18:32:57.4028082Z DASHBOARD_TAG= 2024-05-19T18:32:57.4028396Z GITHUB_RUN_ID=9148947323 2024-05-19T18:32:57.4028855Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.2-1 2024-05-19T18:32:57.4029389Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-05-19T18:32:57.4030440Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.4031519Z GITHUB_ACTOR=pytorch-bot[bot] 2024-05-19T18:32:57.4031931Z NV_LIBNPP_DEV_VERSION=12.2.5.2-1 2024-05-19T18:32:57.4032305Z PR_NUMBER= 2024-05-19T18:32:57.4032604Z GITHUB_RUN_ATTEMPT=1 2024-05-19T18:32:57.4032932Z VALGRIND=ON 2024-05-19T18:32:57.4033231Z ANACONDA_PYTHON_VERSION=3.10 2024-05-19T18:32:57.4033682Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-05-19T18:32:57.4034146Z TERM=xterm 2024-05-19T18:32:57.4034489Z NV_LIBCUSPARSE_DEV_VERSION=12.3.0.142-1 2024-05-19T18:32:57.4034913Z INSTALLED_VISION=yes 2024-05-19T18:32:57.4035234Z BRANCH= 2024-05-19T18:32:57.4035528Z OPENSSL_ROOT_DIR=/opt/openssl 2024-05-19T18:32:57.4035927Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-05-19T18:32:57.4036364Z CUDA_PATH=/usr/local/cuda 2024-05-19T18:32:57.4037116Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-05-19T18:32:57.4038053Z GITHUB_SERVER_URL=https://github.com 2024-05-19T18:32:57.4038542Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-05-19T18:32:57.4039025Z REENABLED_ISSUES= 2024-05-19T18:32:57.4039350Z SHLVL=1 2024-05-19T18:32:57.4039664Z MAX_JOBS=14 2024-05-19T18:32:57.4039983Z NV_CUDA_LIB_VERSION=12.4.0-1 2024-05-19T18:32:57.4040344Z NVARCH=x86_64 2024-05-19T18:32:57.4040656Z GITHUB_ACTOR_ID=54816060 2024-05-19T18:32:57.4041125Z GITHUB_WORKFLOW_SHA=c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T18:32:57.4041687Z GITHUB_REF_NAME=ciflow/inductor/121956 2024-05-19T18:32:57.4042184Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-05-19T18:32:57.4042851Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-05-19T18:32:57.4043436Z GITHUB_JOB=test 2024-05-19T18:32:57.4043845Z NV_LIBNCCL_PACKAGE=libnccl2=2.20.5-1+cuda12.4 2024-05-19T18:32:57.4044409Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-05-19T18:32:57.4044938Z NO_TEST_TIMEOUT=False 2024-05-19T18:32:57.4045270Z TD_DISTRIBUTED=False 2024-05-19T18:32:57.4045767Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.0-1 2024-05-19T18:32:57.4046223Z GITHUB_REPOSITORY=pytorch/pytorch 2024-05-19T18:32:57.4046648Z NV_NVPROF_VERSION=12.4.99-1 2024-05-19T18:32:57.4047010Z GITHUB_RETENTION_DAYS=90 2024-05-19T18:32:57.4047370Z OPENSSL_DIR=/opt/openssl 2024-05-19T18:32:57.4047730Z GITHUB_ACTION_REPOSITORY= 2024-05-19T18:32:57.4048774Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:32:57.4049860Z GITHUB_BASE_REF= 2024-05-19T18:32:57.4050194Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-05-19T18:32:57.4050572Z CI=true 2024-05-19T18:32:57.4050881Z NV_LIBNCCL_PACKAGE_VERSION=2.20.5-1 2024-05-19T18:32:57.4051305Z GITHUB_REPOSITORY_OWNER=pytorch 2024-05-19T18:32:57.4051687Z JOB_ID=25153197363 2024-05-19T18:32:57.4052008Z INSTALLED_PROTOBUF=yes 2024-05-19T18:32:57.4052347Z GITHUB_HEAD_REF= 2024-05-19T18:32:57.4052664Z GITHUB_ACTION_REF= 2024-05-19T18:32:57.4053112Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-05-19T18:32:57.4053582Z GITHUB_WORKFLOW=inductor 2024-05-19T18:32:57.4054033Z DEBIAN_FRONTEND=noninteractive 2024-05-19T18:32:57.4054944Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6f72915a-ad1d-401a-b27b-9b8807dcc300 2024-05-19T18:32:57.4055772Z NO_TD=False 2024-05-19T18:32:57.4056081Z SKIP_SCCACHE_INITIALIZATION=1 2024-05-19T18:32:57.4056456Z _=/usr/bin/env 2024-05-19T18:32:57.4056803Z + echo 'Testing pytorch' 2024-05-19T18:32:57.4057172Z Testing pytorch 2024-05-19T18:32:57.4057508Z + export LANG=C.UTF-8 2024-05-19T18:32:57.4057862Z + LANG=C.UTF-8 2024-05-19T18:32:57.4058166Z + PR_NUMBER= 2024-05-19T18:32:57.4058524Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-05-19T18:32:57.4059083Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-05-19T18:32:57.4059617Z + [[ inductor_torchbench == \s\l\o\w ]] 2024-05-19T18:32:57.4060341Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-05-19T18:32:57.4061091Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]] 2024-05-19T18:32:57.4061654Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:32:57.4062134Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-05-19T18:32:57.4062594Z + [[ inductor_torchbench == *crossref* ]] 2024-05-19T18:32:57.4063159Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-05-19T18:32:57.4063803Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-05-19T18:32:57.4064464Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:32:57.4065027Z + pip_install --user ninja==1.10.2 2024-05-19T18:32:57.4065554Z + pip install --progress-bar off --user ninja==1.10.2 2024-05-19T18:32:57.7563484Z Collecting ninja==1.10.2 2024-05-19T18:32:57.7764670Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-05-19T18:32:57.7912022Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-05-19T18:32:59.0664537Z Installing collected packages: ninja 2024-05-19T18:32:59.0732109Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-05-19T18:32:59.0733393Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-05-19T18:32:59.0771033Z Successfully installed ninja-1.10.2 2024-05-19T18:32:59.1629292Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:32:59.1631554Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:32:59.1632961Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:32:59.1633468Z + install_tlparse 2024-05-19T18:32:59.1634058Z + pip_install --user tlparse==0.3.7 2024-05-19T18:32:59.1634618Z + pip install --progress-bar off --user tlparse==0.3.7 2024-05-19T18:32:59.5089674Z Collecting tlparse==0.3.7 2024-05-19T18:32:59.5341948Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-05-19T18:32:59.5449239Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-05-19T18:33:00.8322534Z Installing collected packages: tlparse 2024-05-19T18:33:00.8682447Z Successfully installed tlparse-0.3.7 2024-05-19T18:33:00.9512029Z ++ python -m site --user-base 2024-05-19T18:33:00.9667852Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-05-19T18:33:00.9671073Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]] 2024-05-19T18:33:00.9671774Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]] 2024-05-19T18:33:00.9672447Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-05-19T18:33:00.9673675Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-05-19T18:33:00.9674833Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass 2024-05-19T18:33:00.9675541Z + cd test 2024-05-19T18:33:00.9676077Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-05-19T18:33:02.3502699Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-05-19T18:33:02.3503333Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-05-19T18:33:02.3505279Z + DYNAMO_BENCHMARK_FLAGS=() 2024-05-19T18:33:02.3505922Z + [[ inductor_torchbench == *dynamo_eager* ]] 2024-05-19T18:33:02.3506425Z + [[ inductor_torchbench == *aot_eager* ]] 2024-05-19T18:33:02.3506917Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-05-19T18:33:02.3507418Z + [[ inductor_torchbench == *inductor* ]] 2024-05-19T18:33:02.3507869Z + [[ inductor_torchbench != *perf* ]] 2024-05-19T18:33:02.3508481Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-05-19T18:33:02.3508929Z + [[ inductor_torchbench == *dynamic* ]] 2024-05-19T18:33:02.3509403Z + [[ inductor_torchbench == *cpu_inductor* ]] 2024-05-19T18:33:02.3509919Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-05-19T18:33:02.3519638Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *tbb* ]] 2024-05-19T18:33:02.3535474Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:33:02.3536168Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-05-19T18:33:02.3537728Z + cd test 2024-05-19T18:33:02.3538274Z + python -c 'import torch; print(torch.__config__.show())' 2024-05-19T18:33:03.5735920Z PyTorch built with: 2024-05-19T18:33:03.5736511Z - GCC 9.4 2024-05-19T18:33:03.5736975Z - C++ Version: 201703 2024-05-19T18:33:03.5737968Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:33:03.5739060Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:33:03.5739736Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:33:03.5740242Z - LAPACK is enabled (usually provided by MKL) 2024-05-19T18:33:03.5740726Z - NNPACK is enabled 2024-05-19T18:33:03.5741105Z - CPU capability usage: AVX2 2024-05-19T18:33:03.5741508Z - CUDA Runtime 12.4 2024-05-19T18:33:03.5742014Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-05-19T18:33:03.5742617Z - CuDNN 8.9.7 (built against CUDA 12.2) 2024-05-19T18:33:03.5743061Z - Magma 2.6.1 2024-05-19T18:33:03.5750376Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=8.9.7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-05-19T18:33:03.5757149Z 2024-05-19T18:33:03.8431138Z + cd test 2024-05-19T18:33:03.8437319Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-05-19T18:33:04.9547995Z ATen/Parallel: 2024-05-19T18:33:04.9548634Z at::get_num_threads() : 8 2024-05-19T18:33:04.9549217Z at::get_num_interop_threads() : 16 2024-05-19T18:33:04.9549808Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-05-19T18:33:04.9550357Z omp_get_max_threads() : 8 2024-05-19T18:33:04.9551827Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-05-19T18:33:04.9553018Z mkl_get_max_threads() : 8 2024-05-19T18:33:04.9553915Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-05-19T18:33:04.9554876Z std::thread::hardware_concurrency() : 16 2024-05-19T18:33:04.9555480Z Environment variables: 2024-05-19T18:33:04.9555968Z OMP_NUM_THREADS : [not set] 2024-05-19T18:33:04.9556489Z MKL_NUM_THREADS : [not set] 2024-05-19T18:33:04.9557052Z ATen parallel backend: OpenMP 2024-05-19T18:33:04.9557458Z 2024-05-19T18:33:05.1938220Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-05-19T18:33:05.1938926Z + [[ inductor_torchbench == *backward* ]] 2024-05-19T18:33:05.1939551Z + [[ inductor_torchbench == *xla* ]] 2024-05-19T18:33:05.1940032Z + [[ inductor_torchbench == *executorch* ]] 2024-05-19T18:33:05.1940539Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-05-19T18:33:05.1941198Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-05-19T18:33:05.1941941Z + [[ inductor_torchbench == distributed ]] 2024-05-19T18:33:05.1942526Z + [[ inductor_torchbench == deploy ]] 2024-05-19T18:33:05.1943153Z + [[ inductor_torchbench == *inductor_distributed* ]] 2024-05-19T18:33:05.1944035Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]] 2024-05-19T18:33:05.1944775Z + [[ inductor_torchbench == *huggingface* ]] 2024-05-19T18:33:05.1945440Z + [[ inductor_torchbench == *timm* ]] 2024-05-19T18:33:05.1946062Z + [[ inductor_torchbench == *torchbench* ]] 2024-05-19T18:33:05.1946733Z + [[ inductor_torchbench == *cpu_inductor* ]] 2024-05-19T18:33:05.1947335Z + install_torchaudio cuda 2024-05-19T18:33:05.1947799Z + local commit 2024-05-19T18:33:05.1948252Z ++ get_pinned_commit audio 2024-05-19T18:33:05.1948814Z ++ cat .github/ci_commit_pins/audio.txt 2024-05-19T18:33:05.1955158Z + commit=1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:33:05.1955634Z + [[ cuda == \c\u\d\a ]] 2024-05-19T18:33:05.1956061Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-05-19T18:33:05.1957067Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:33:05.1958918Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:33:05.5048101Z Collecting git+https://github.com/pytorch/audio.git@1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:33:05.5051200Z Cloning https://github.com/pytorch/audio.git (to revision 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0) to /tmp/pip-req-build-crh60t34 2024-05-19T18:33:05.5068140Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-crh60t34 2024-05-19T18:33:06.2225833Z Running command git rev-parse -q --verify 'sha^1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0' 2024-05-19T18:33:06.2244924Z Running command git fetch -q https://github.com/pytorch/audio.git 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:33:06.6214333Z Resolved https://github.com/pytorch/audio.git to commit 1980f8af5bcd0bb2ce51965cf79d8d4c25dad8a0 2024-05-19T18:33:06.6216066Z Running command git submodule update --init --recursive -q 2024-05-19T18:33:08.6473756Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:33:08.6504576Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.2.0a0+1980f8a) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:33:08.6568088Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.13.1) 2024-05-19T18:33:08.6573892Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (4.11.0) 2024-05-19T18:33:08.6577100Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (1.12) 2024-05-19T18:33:08.6580331Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2.8.8) 2024-05-19T18:33:08.6583746Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (3.1.4) 2024-05-19T18:33:08.6587017Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.2.0a0+1980f8a) (2024.2.0) 2024-05-19T18:33:08.7006044Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.2.0a0+1980f8a) (2.1.5) 2024-05-19T18:33:08.7160751Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.2.0a0+1980f8a) (1.3.0) 2024-05-19T18:33:08.7229563Z Building wheels for collected packages: torchaudio 2024-05-19T18:35:24.9640614Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-05-19T18:35:24.9668659Z [?25h Created wheel for torchaudio: filename=torchaudio-2.2.0a0+1980f8a-cp310-cp310-linux_x86_64.whl size=2311968 sha256=7a3114dcd5010cf4f4c35a84382d653efad866cc84be97cbf399a03f0b8db292 2024-05-19T18:35:24.9671168Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3c/69/04/2f8953929b90f7f98a00671a3b24cfeb70fc70e57328823e5c 2024-05-19T18:35:24.9697731Z Successfully built torchaudio 2024-05-19T18:35:26.1666460Z Installing collected packages: torchaudio 2024-05-19T18:35:26.3576928Z Successfully installed torchaudio-2.2.0a0+1980f8a 2024-05-19T18:35:26.4881795Z + install_torchtext 2024-05-19T18:35:26.4883661Z + local data_commit 2024-05-19T18:35:26.4884353Z + local text_commit 2024-05-19T18:35:26.4884826Z ++ get_pinned_commit data 2024-05-19T18:35:26.4885354Z ++ cat .github/ci_commit_pins/data.txt 2024-05-19T18:35:26.4895718Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:35:26.4897808Z ++ get_pinned_commit text 2024-05-19T18:35:26.4898325Z ++ cat .github/ci_commit_pins/text.txt 2024-05-19T18:35:26.4907658Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:35:26.4908865Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:35:26.4910308Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:35:26.8054023Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:35:26.8057612Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-42okewp1 2024-05-19T18:35:26.8076152Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-42okewp1 2024-05-19T18:35:27.3003463Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-05-19T18:35:27.3017645Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:35:27.5433787Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:35:27.7100329Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-05-19T18:35:27.7101637Z Running command git submodule update --init --recursive -q 2024-05-19T18:37:20.6401472Z fatal: the remote end hung up unexpectedly 2024-05-19T18:37:20.6403714Z fatal: early EOF 2024-05-19T18:37:20.6404316Z fatal: index-pack failed 2024-05-19T18:37:20.6424204Z fatal: clone of 'https://github.com/awslabs/aws-c-io.git' into submodule path '/tmp/pip-req-build-42okewp1/third_party/aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-io' failed 2024-05-19T18:37:20.6425710Z Failed to clone 'crt/aws-c-io'. Retry scheduled 2024-05-19T18:38:05.7234974Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:38:05.7287586Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.18) 2024-05-19T18:38:05.7290680Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.31.0) 2024-05-19T18:38:05.7296419Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:38:05.7357151Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-05-19T18:38:05.7362588Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.11.0) 2024-05-19T18:38:05.7365441Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12) 2024-05-19T18:38:05.7368719Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-05-19T18:38:05.7371693Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-05-19T18:38:05.7374854Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-05-19T18:38:05.7560302Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-05-19T18:38:05.7563975Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-05-19T18:38:05.7571323Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.2.2) 2024-05-19T18:38:05.8022187Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-05-19T18:38:05.8184218Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-05-19T18:38:05.8265140Z Building wheels for collected packages: torchdata 2024-05-19T18:38:07.4358750Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-05-19T18:38:07.4366044Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=d16c2896c485a930b19e9dca9ff73069987cae70a11fa7be249887008d80c239 2024-05-19T18:38:07.4368713Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-05-19T18:38:07.4396261Z Successfully built torchdata 2024-05-19T18:38:08.6194358Z Installing collected packages: torchdata 2024-05-19T18:38:08.7008623Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-05-19T18:38:11.2127470Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:11.2129314Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:11.5230738Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:11.5233843Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-_eatj338 2024-05-19T18:38:11.5247127Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-_eatj338 2024-05-19T18:38:51.9058108Z error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:38:51.9059617Z fatal: the remote end hung up unexpectedly 2024-05-19T18:38:51.9060115Z fatal: early EOF 2024-05-19T18:38:51.9063924Z fatal: index-pack failed 2024-05-19T18:38:51.9114514Z error: subprocess-exited-with-error 2024-05-19T18:38:51.9115048Z 2024-05-19T18:38:51.9118173Z × git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-_eatj338 did not run successfully. 2024-05-19T18:38:51.9119781Z │ exit code: 128 2024-05-19T18:38:51.9120257Z ╰─> See above for output. 2024-05-19T18:38:51.9120639Z 2024-05-19T18:38:51.9121272Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:38:51.9128464Z error: subprocess-exited-with-error 2024-05-19T18:38:51.9128845Z 2024-05-19T18:38:51.9130125Z × git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-_eatj338 did not run successfully. 2024-05-19T18:38:51.9131455Z │ exit code: 128 2024-05-19T18:38:51.9131922Z ╰─> See above for output. 2024-05-19T18:38:51.9132212Z 2024-05-19T18:38:51.9132697Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:38:51.9600489Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:52.2705902Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:52.2710094Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-6oafee8j 2024-05-19T18:38:52.2728032Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-6oafee8j 2024-05-19T18:38:53.0653132Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-05-19T18:38:53.0671362Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:53.5300718Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:53.7061837Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-05-19T18:38:53.7062958Z Running command git submodule update --init --recursive -q 2024-05-19T18:39:17.0153804Z fatal: unable to access 'https://github.com/google/re2/': GnuTLS recv error (-9): Error decoding the received TLS packet. 2024-05-19T18:39:17.0167715Z fatal: clone of 'https://github.com/google/re2' into submodule path '/tmp/pip-req-build-6oafee8j/third_party/re2' failed 2024-05-19T18:39:17.0169011Z Failed to clone 'third_party/re2'. Retry scheduled 2024-05-19T18:39:22.6075177Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:39:22.6126151Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-05-19T18:39:22.6128932Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.31.0) 2024-05-19T18:39:22.6132216Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:39:22.6135291Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-05-19T18:39:22.6138207Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-05-19T18:39:22.6198332Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-05-19T18:39:22.6205217Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-05-19T18:39:22.6210236Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.18) 2024-05-19T18:39:22.6215401Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.2.2) 2024-05-19T18:39:22.6272160Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-05-19T18:39:22.6276535Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.11.0) 2024-05-19T18:39:22.6279493Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12) 2024-05-19T18:39:22.6283061Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-05-19T18:39:22.6285954Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-05-19T18:39:22.6288728Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-05-19T18:39:22.6965794Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-05-19T18:39:22.7124760Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-05-19T18:39:22.7202895Z Building wheels for collected packages: torchtext 2024-05-19T18:39:59.7182207Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-05-19T18:39:59.7208704Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045366 sha256=e7bb2fe326a79ef045781b7ec76db14834d641ec0b21a2eeac930e3917d8a496 2024-05-19T18:39:59.7210594Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-05-19T18:39:59.7233265Z Successfully built torchtext 2024-05-19T18:40:00.8913870Z Installing collected packages: torchtext 2024-05-19T18:40:00.9888204Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-05-19T18:40:01.1140285Z + install_torchvision 2024-05-19T18:40:01.1140684Z + local orig_preload 2024-05-19T18:40:01.1141022Z + local commit 2024-05-19T18:40:01.1142601Z ++ get_pinned_commit vision 2024-05-19T18:40:01.1143972Z ++ cat .github/ci_commit_pins/vision.txt 2024-05-19T18:40:01.1152241Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:40:01.1152911Z + orig_preload= 2024-05-19T18:40:01.1153407Z + '[' -n '' ']' 2024-05-19T18:40:01.1154558Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:40:01.1156251Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:40:01.4341655Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:40:01.4344955Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-n_r29dxv 2024-05-19T18:40:01.4361736Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-n_r29dxv 2024-05-19T18:40:02.7326170Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-05-19T18:40:02.7339635Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:40:03.9385217Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:40:04.1503087Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-05-19T18:40:06.2631689Z Preparing metadata (setup.py) ... [?25l- \ done 2024-05-19T18:40:06.2681735Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-05-19T18:40:06.2685802Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.4.0a0+gitc8c7ddf) 2024-05-19T18:40:06.2692593Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-05-19T18:40:06.2900556Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-05-19T18:40:06.2907743Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.11.0) 2024-05-19T18:40:06.2910555Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12) 2024-05-19T18:40:06.2914670Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-05-19T18:40:06.2918483Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-05-19T18:40:06.2922621Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-05-19T18:40:06.3339322Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-05-19T18:40:06.3489562Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-05-19T18:40:06.3563169Z Building wheels for collected packages: torchvision 2024-05-19T18:41:16.2448982Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-05-19T18:41:16.2472501Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2025714 sha256=a8e46c1f4fbefc5a23f0e420abb1c5aa43d5890777a5dd72012e0af40d1ce740 2024-05-19T18:41:16.2474191Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-05-19T18:41:16.2508880Z Successfully built torchvision 2024-05-19T18:41:17.4418091Z Installing collected packages: torchvision 2024-05-19T18:41:17.8219317Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-05-19T18:41:17.9351463Z + '[' -n '' ']' 2024-05-19T18:41:17.9351850Z + id=0 2024-05-19T18:41:17.9352218Z + pip_install opencv-python==4.8.0.74 2024-05-19T18:41:17.9352799Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-05-19T18:41:18.3684275Z Collecting opencv-python==4.8.0.74 2024-05-19T18:41:18.3878826Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-05-19T18:41:18.4055556Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-05-19T18:41:18.4161273Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-05-19T18:41:20.2332871Z Installing collected packages: opencv-python 2024-05-19T18:41:21.0519777Z Successfully installed opencv-python-4.8.0.74 2024-05-19T18:41:21.1515765Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-05-19T18:41:21.1516656Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-05-19T18:41:21.1517408Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-05-19T18:41:21.1518214Z + checkout_install_torchbench 2024-05-19T18:41:21.1518584Z + local commit 2024-05-19T18:41:21.1518914Z ++ get_pinned_commit torchbench 2024-05-19T18:41:21.1519337Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-05-19T18:41:21.1528425Z + commit=d6015d42d9a1834bc7595c4bd6852562fb80b30b 2024-05-19T18:41:21.1529077Z + git clone https://github.com/pytorch/benchmark torchbench 2024-05-19T18:41:21.1539178Z Cloning into 'torchbench'... 2024-05-19T18:41:21.2944822Z remote: Enumerating objects: 29198, done. 2024-05-19T18:41:21.2946261Z remote: Counting objects: 0% (1/4121) 2024-05-19T18:41:21.2947733Z remote: Counting objects: 1% (42/4121) 2024-05-19T18:41:21.2949168Z remote: Counting objects: 2% (83/4121) 2024-05-19T18:41:21.2950662Z remote: Counting objects: 3% (124/4121) 2024-05-19T18:41:21.2952181Z remote: Counting objects: 4% (165/4121) 2024-05-19T18:41:21.2953000Z remote: Counting objects: 5% (207/4121) 2024-05-19T18:41:21.2953748Z remote: Counting objects: 6% (248/4121) 2024-05-19T18:41:21.2954423Z remote: Counting objects: 7% (289/4121) 2024-05-19T18:41:21.2955091Z remote: Counting objects: 8% (330/4121) 2024-05-19T18:41:21.2955807Z remote: Counting objects: 9% (371/4121) 2024-05-19T18:41:21.2956480Z remote: Counting objects: 10% (413/4121) 2024-05-19T18:41:21.2957142Z remote: Counting objects: 11% (454/4121) 2024-05-19T18:41:21.2957964Z remote: Counting objects: 12% (495/4121) 2024-05-19T18:41:21.2958742Z remote: Counting objects: 13% (536/4121) 2024-05-19T18:41:21.2959364Z remote: Counting objects: 14% (577/4121) 2024-05-19T18:41:21.2960041Z remote: Counting objects: 15% (619/4121) 2024-05-19T18:41:21.2960560Z remote: Counting objects: 16% (660/4121) 2024-05-19T18:41:21.2961144Z remote: Counting objects: 17% (701/4121) 2024-05-19T18:41:21.2961826Z remote: Counting objects: 18% (742/4121) 2024-05-19T18:41:21.2962534Z remote: Counting objects: 19% (783/4121) 2024-05-19T18:41:21.2963165Z remote: Counting objects: 20% (825/4121) 2024-05-19T18:41:21.2963678Z remote: Counting objects: 21% (866/4121) 2024-05-19T18:41:21.2964193Z remote: Counting objects: 22% (907/4121) 2024-05-19T18:41:21.2964894Z remote: Counting objects: 23% (948/4121) 2024-05-19T18:41:21.2965523Z remote: Counting objects: 24% (990/4121) 2024-05-19T18:41:21.2966129Z remote: Counting objects: 25% (1031/4121) 2024-05-19T18:41:21.2966878Z remote: Counting objects: 26% (1072/4121) 2024-05-19T18:41:21.2967458Z remote: Counting objects: 27% (1113/4121) 2024-05-19T18:41:21.2968206Z remote: Counting objects: 28% (1154/4121) 2024-05-19T18:41:21.2968772Z remote: Counting objects: 29% (1196/4121) 2024-05-19T18:41:21.2969340Z remote: Counting objects: 30% (1237/4121) 2024-05-19T18:41:21.2969932Z remote: Counting objects: 31% (1278/4121) 2024-05-19T18:41:21.2970448Z remote: Counting objects: 32% (1319/4121) 2024-05-19T18:41:21.2970961Z remote: Counting objects: 33% (1360/4121) 2024-05-19T18:41:21.2971473Z remote: Counting objects: 34% (1402/4121) 2024-05-19T18:41:21.2971981Z remote: Counting objects: 35% (1443/4121) 2024-05-19T18:41:21.2972493Z remote: Counting objects: 36% (1484/4121) 2024-05-19T18:41:21.2972996Z remote: Counting objects: 37% (1525/4121) 2024-05-19T18:41:21.2973502Z remote: Counting objects: 38% (1566/4121) 2024-05-19T18:41:21.2974048Z remote: Counting objects: 39% (1608/4121) 2024-05-19T18:41:21.2974637Z remote: Counting objects: 40% (1649/4121) 2024-05-19T18:41:21.2975242Z remote: Counting objects: 41% (1690/4121) 2024-05-19T18:41:21.2975872Z remote: Counting objects: 42% (1731/4121) 2024-05-19T18:41:21.2976386Z remote: Counting objects: 43% (1773/4121) 2024-05-19T18:41:21.2976889Z remote: Counting objects: 44% (1814/4121) 2024-05-19T18:41:21.2977400Z remote: Counting objects: 45% (1855/4121) 2024-05-19T18:41:21.2977911Z remote: Counting objects: 46% (1896/4121) 2024-05-19T18:41:21.2978420Z remote: Counting objects: 47% (1937/4121) 2024-05-19T18:41:21.2978921Z remote: Counting objects: 48% (1979/4121) 2024-05-19T18:41:21.2979427Z remote: Counting objects: 49% (2020/4121) 2024-05-19T18:41:21.2979933Z remote: Counting objects: 50% (2061/4121) 2024-05-19T18:41:21.2980443Z remote: Counting objects: 51% (2102/4121) 2024-05-19T18:41:21.2980944Z remote: Counting objects: 52% (2143/4121) 2024-05-19T18:41:21.2981455Z remote: Counting objects: 53% (2185/4121) 2024-05-19T18:41:21.2981968Z remote: Counting objects: 54% (2226/4121) 2024-05-19T18:41:21.2982477Z remote: Counting objects: 55% (2267/4121) 2024-05-19T18:41:21.2982985Z remote: Counting objects: 56% (2308/4121) 2024-05-19T18:41:21.2983495Z remote: Counting objects: 57% (2349/4121) 2024-05-19T18:41:21.2984002Z remote: Counting objects: 58% (2391/4121) 2024-05-19T18:41:21.2984508Z remote: Counting objects: 59% (2432/4121) 2024-05-19T18:41:21.2985008Z remote: Counting objects: 60% (2473/4121) 2024-05-19T18:41:21.2985513Z remote: Counting objects: 61% (2514/4121) 2024-05-19T18:41:21.2986014Z remote: Counting objects: 62% (2556/4121) 2024-05-19T18:41:21.2986522Z remote: Counting objects: 63% (2597/4121) 2024-05-19T18:41:21.2987020Z remote: Counting objects: 64% (2638/4121) 2024-05-19T18:41:21.2987526Z remote: Counting objects: 65% (2679/4121) 2024-05-19T18:41:21.2988033Z remote: Counting objects: 66% (2720/4121) 2024-05-19T18:41:21.2988531Z remote: Counting objects: 67% (2762/4121) 2024-05-19T18:41:21.2989123Z remote: Counting objects: 68% (2803/4121) 2024-05-19T18:41:21.2989638Z remote: Counting objects: 69% (2844/4121) 2024-05-19T18:41:21.2990143Z remote: Counting objects: 70% (2885/4121) 2024-05-19T18:41:21.2990644Z remote: Counting objects: 71% (2926/4121) 2024-05-19T18:41:21.2991261Z remote: Counting objects: 72% (2968/4121) 2024-05-19T18:41:21.2991784Z remote: Counting objects: 73% (3009/4121) 2024-05-19T18:41:21.2992355Z remote: Counting objects: 74% (3050/4121) 2024-05-19T18:41:21.2992973Z remote: Counting objects: 75% (3091/4121) 2024-05-19T18:41:21.2993482Z remote: Counting objects: 76% (3132/4121) 2024-05-19T18:41:21.2994073Z remote: Counting objects: 77% (3174/4121) 2024-05-19T18:41:21.2994586Z remote: Counting objects: 78% (3215/4121) 2024-05-19T18:41:21.2995088Z remote: Counting objects: 79% (3256/4121) 2024-05-19T18:41:21.2995596Z remote: Counting objects: 80% (3297/4121) 2024-05-19T18:41:21.2996107Z remote: Counting objects: 81% (3339/4121) 2024-05-19T18:41:21.2996613Z remote: Counting objects: 82% (3380/4121) 2024-05-19T18:41:21.2997197Z remote: Counting objects: 83% (3421/4121) 2024-05-19T18:41:21.2997852Z remote: Counting objects: 84% (3462/4121) 2024-05-19T18:41:21.2998363Z remote: Counting objects: 85% (3503/4121) 2024-05-19T18:41:21.2998872Z remote: Counting objects: 86% (3545/4121) 2024-05-19T18:41:21.2999371Z remote: Counting objects: 87% (3586/4121) 2024-05-19T18:41:21.2999882Z remote: Counting objects: 88% (3627/4121) 2024-05-19T18:41:21.3000392Z remote: Counting objects: 89% (3668/4121) 2024-05-19T18:41:21.3000899Z remote: Counting objects: 90% (3709/4121) 2024-05-19T18:41:21.3001400Z remote: Counting objects: 91% (3751/4121) 2024-05-19T18:41:21.3001906Z remote: Counting objects: 92% (3792/4121) 2024-05-19T18:41:21.3002413Z remote: Counting objects: 93% (3833/4121) 2024-05-19T18:41:21.3002923Z remote: Counting objects: 94% (3874/4121) 2024-05-19T18:41:21.3003426Z remote: Counting objects: 95% (3915/4121) 2024-05-19T18:41:21.3003932Z remote: Counting objects: 96% (3957/4121) 2024-05-19T18:41:21.3004442Z remote: Counting objects: 97% (3998/4121) 2024-05-19T18:41:21.3004940Z remote: Counting objects: 98% (4039/4121) 2024-05-19T18:41:21.3005447Z remote: Counting objects: 99% (4080/4121) 2024-05-19T18:41:21.3005953Z remote: Counting objects: 100% (4121/4121) 2024-05-19T18:41:21.3006496Z remote: Counting objects: 100% (4121/4121), done. 2024-05-19T18:41:21.3007042Z remote: Compressing objects: 0% (1/832) 2024-05-19T18:41:21.3007570Z remote: Compressing objects: 1% (9/832) 2024-05-19T18:41:21.3013267Z remote: Compressing objects: 2% (17/832) 2024-05-19T18:41:21.3060005Z remote: Compressing objects: 3% (25/832) 2024-05-19T18:41:21.3063379Z remote: Compressing objects: 4% (34/832) 2024-05-19T18:41:21.3082705Z remote: Compressing objects: 5% (42/832) 2024-05-19T18:41:21.3098297Z remote: Compressing objects: 6% (50/832) 2024-05-19T18:41:21.3106431Z remote: Compressing objects: 7% (59/832) 2024-05-19T18:41:21.3121830Z remote: Compressing objects: 8% (67/832) 2024-05-19T18:41:21.3138125Z remote: Compressing objects: 9% (75/832) 2024-05-19T18:41:21.3144839Z remote: Compressing objects: 10% (84/832) 2024-05-19T18:41:21.3156251Z remote: Compressing objects: 11% (92/832) 2024-05-19T18:41:21.3271072Z remote: Compressing objects: 12% (100/832) 2024-05-19T18:41:21.3311607Z remote: Compressing objects: 13% (109/832) 2024-05-19T18:41:21.3368202Z remote: Compressing objects: 14% (117/832) 2024-05-19T18:41:21.3454728Z remote: Compressing objects: 15% (125/832) 2024-05-19T18:41:21.3500265Z remote: Compressing objects: 16% (134/832) 2024-05-19T18:41:21.3569696Z remote: Compressing objects: 17% (142/832) 2024-05-19T18:41:21.3630552Z remote: Compressing objects: 18% (150/832) 2024-05-19T18:41:21.3669262Z remote: Compressing objects: 19% (159/832) 2024-05-19T18:41:21.3724792Z remote: Compressing objects: 20% (167/832) 2024-05-19T18:41:21.3768717Z remote: Compressing objects: 21% (175/832) 2024-05-19T18:41:21.3813143Z remote: Compressing objects: 22% (184/832) 2024-05-19T18:41:21.3845773Z remote: Compressing objects: 23% (192/832) 2024-05-19T18:41:21.3866020Z remote: Compressing objects: 24% (200/832) 2024-05-19T18:41:21.3897995Z remote: Compressing objects: 25% (208/832) 2024-05-19T18:41:21.3933215Z remote: Compressing objects: 26% (217/832) 2024-05-19T18:41:21.3955289Z remote: Compressing objects: 27% (225/832) 2024-05-19T18:41:21.3977014Z remote: Compressing objects: 28% (233/832) 2024-05-19T18:41:21.3997045Z remote: Compressing objects: 29% (242/832) 2024-05-19T18:41:21.4011940Z remote: Compressing objects: 30% (250/832) 2024-05-19T18:41:21.4037265Z remote: Compressing objects: 31% (258/832) 2024-05-19T18:41:21.4050269Z remote: Compressing objects: 32% (267/832) 2024-05-19T18:41:21.4063401Z remote: Compressing objects: 33% (275/832) 2024-05-19T18:41:21.4076983Z remote: Compressing objects: 34% (283/832) 2024-05-19T18:41:21.4084546Z remote: Compressing objects: 35% (292/832) 2024-05-19T18:41:21.4091391Z remote: Compressing objects: 36% (300/832) 2024-05-19T18:41:21.4095467Z remote: Compressing objects: 37% (308/832) 2024-05-19T18:41:21.4098165Z remote: Compressing objects: 38% (317/832) 2024-05-19T18:41:21.4099102Z remote: Compressing objects: 39% (325/832) 2024-05-19T18:41:21.4100168Z remote: Compressing objects: 40% (333/832) 2024-05-19T18:41:21.4101990Z remote: Compressing objects: 41% (342/832) 2024-05-19T18:41:21.4102603Z remote: Compressing objects: 42% (350/832) 2024-05-19T18:41:21.4105138Z remote: Compressing objects: 43% (358/832) 2024-05-19T18:41:21.4105691Z remote: Compressing objects: 44% (367/832) 2024-05-19T18:41:21.4107930Z remote: Compressing objects: 45% (375/832) 2024-05-19T18:41:21.4109661Z remote: Compressing objects: 46% (383/832) 2024-05-19T18:41:21.4115279Z remote: Compressing objects: 47% (392/832) 2024-05-19T18:41:21.4117082Z remote: Compressing objects: 48% (400/832) 2024-05-19T18:41:21.4119744Z remote: Compressing objects: 49% (408/832) 2024-05-19T18:41:21.4122880Z remote: Compressing objects: 50% (416/832) 2024-05-19T18:41:21.4125446Z remote: Compressing objects: 51% (425/832) 2024-05-19T18:41:21.4128059Z remote: Compressing objects: 52% (433/832) 2024-05-19T18:41:21.4130912Z remote: Compressing objects: 53% (441/832) 2024-05-19T18:41:21.4132515Z remote: Compressing objects: 54% (450/832) 2024-05-19T18:41:21.4136139Z remote: Compressing objects: 55% (458/832) 2024-05-19T18:41:21.4139365Z remote: Compressing objects: 56% (466/832) 2024-05-19T18:41:21.4142487Z remote: Compressing objects: 57% (475/832) 2024-05-19T18:41:21.4145423Z remote: Compressing objects: 58% (483/832) 2024-05-19T18:41:21.4149062Z remote: Compressing objects: 59% (491/832) 2024-05-19T18:41:21.4150243Z remote: Compressing objects: 60% (500/832) 2024-05-19T18:41:21.4153832Z remote: Compressing objects: 61% (508/832) 2024-05-19T18:41:21.4157315Z remote: Compressing objects: 62% (516/832) 2024-05-19T18:41:21.4159619Z remote: Compressing objects: 63% (525/832) 2024-05-19T18:41:21.4161443Z remote: Compressing objects: 64% (533/832) 2024-05-19T18:41:21.4163434Z remote: Compressing objects: 65% (541/832) 2024-05-19T18:41:21.4166132Z remote: Compressing objects: 66% (550/832) 2024-05-19T18:41:21.4168724Z remote: Compressing objects: 67% (558/832) 2024-05-19T18:41:21.4169362Z remote: Compressing objects: 68% (566/832) 2024-05-19T18:41:21.4170049Z remote: Compressing objects: 69% (575/832) 2024-05-19T18:41:21.4171592Z remote: Compressing objects: 70% (583/832) 2024-05-19T18:41:21.4172320Z remote: Compressing objects: 71% (591/832) 2024-05-19T18:41:21.4173336Z remote: Compressing objects: 72% (600/832) 2024-05-19T18:41:21.4174783Z remote: Compressing objects: 73% (608/832) 2024-05-19T18:41:21.4175614Z remote: Compressing objects: 74% (616/832) 2024-05-19T18:41:21.4176173Z remote: Compressing objects: 75% (624/832) 2024-05-19T18:41:21.4177235Z remote: Compressing objects: 76% (633/832) 2024-05-19T18:41:21.4177972Z remote: Compressing objects: 77% (641/832) 2024-05-19T18:41:21.4178590Z remote: Compressing objects: 78% (649/832) 2024-05-19T18:41:21.4180300Z remote: Compressing objects: 79% (658/832) 2024-05-19T18:41:21.4181078Z remote: Compressing objects: 80% (666/832) 2024-05-19T18:41:21.4181752Z remote: Compressing objects: 81% (674/832) 2024-05-19T18:41:21.4182298Z remote: Compressing objects: 82% (683/832) 2024-05-19T18:41:21.4182913Z remote: Compressing objects: 83% (691/832) 2024-05-19T18:41:21.4183614Z remote: Compressing objects: 84% (699/832) 2024-05-19T18:41:21.4184146Z remote: Compressing objects: 85% (708/832) 2024-05-19T18:41:21.4184681Z remote: Compressing objects: 86% (716/832) 2024-05-19T18:41:21.4185325Z remote: Compressing objects: 87% (724/832) 2024-05-19T18:41:21.4186011Z remote: Compressing objects: 88% (733/832) 2024-05-19T18:41:21.4186781Z remote: Compressing objects: 89% (741/832) 2024-05-19T18:41:21.4187503Z remote: Compressing objects: 90% (749/832) 2024-05-19T18:41:21.4188131Z remote: Compressing objects: 91% (758/832) 2024-05-19T18:41:21.4188793Z remote: Compressing objects: 92% (766/832) 2024-05-19T18:41:21.4189472Z remote: Compressing objects: 93% (774/832) 2024-05-19T18:41:21.4190304Z remote: Compressing objects: 94% (783/832) 2024-05-19T18:41:21.4192211Z remote: Compressing objects: 95% (791/832) 2024-05-19T18:41:21.4194858Z remote: Compressing objects: 96% (799/832) 2024-05-19T18:41:21.4195905Z remote: Compressing objects: 97% (808/832) 2024-05-19T18:41:21.4196784Z remote: Compressing objects: 98% (816/832) 2024-05-19T18:41:21.4197441Z remote: Compressing objects: 99% (824/832) 2024-05-19T18:41:21.4198641Z remote: Compressing objects: 100% (832/832) 2024-05-19T18:41:21.4199276Z remote: Compressing objects: 100% (832/832), done. 2024-05-19T18:41:21.4284651Z Receiving objects: 0% (1/29198) 2024-05-19T18:41:21.4328381Z Receiving objects: 1% (292/29198) 2024-05-19T18:41:21.4346897Z Receiving objects: 2% (584/29198) 2024-05-19T18:41:21.4386998Z Receiving objects: 3% (876/29198) 2024-05-19T18:41:21.4443681Z Receiving objects: 4% (1168/29198) 2024-05-19T18:41:21.4544674Z Receiving objects: 5% (1460/29198) 2024-05-19T18:41:21.4568860Z Receiving objects: 6% (1752/29198) 2024-05-19T18:41:21.4663726Z Receiving objects: 7% (2044/29198) 2024-05-19T18:41:21.4731649Z Receiving objects: 8% (2336/29198) 2024-05-19T18:41:21.4766300Z Receiving objects: 9% (2628/29198) 2024-05-19T18:41:21.4794856Z Receiving objects: 10% (2920/29198) 2024-05-19T18:41:21.4825778Z Receiving objects: 11% (3212/29198) 2024-05-19T18:41:21.4859500Z Receiving objects: 12% (3504/29198) 2024-05-19T18:41:21.4897038Z Receiving objects: 13% (3796/29198) 2024-05-19T18:41:21.4932542Z Receiving objects: 14% (4088/29198) 2024-05-19T18:41:21.4962427Z Receiving objects: 15% (4380/29198) 2024-05-19T18:41:21.4989411Z Receiving objects: 16% (4672/29198) 2024-05-19T18:41:21.5032766Z Receiving objects: 17% (4964/29198) 2024-05-19T18:41:21.5086216Z Receiving objects: 18% (5256/29198) 2024-05-19T18:41:21.5204756Z Receiving objects: 19% (5548/29198) 2024-05-19T18:41:21.5358956Z Receiving objects: 20% (5840/29198) 2024-05-19T18:41:21.5404293Z Receiving objects: 21% (6132/29198) 2024-05-19T18:41:21.5434380Z Receiving objects: 22% (6424/29198) 2024-05-19T18:41:21.5448779Z Receiving objects: 23% (6716/29198) 2024-05-19T18:41:21.5474814Z Receiving objects: 24% (7008/29198) 2024-05-19T18:41:21.5504456Z Receiving objects: 25% (7300/29198) 2024-05-19T18:41:21.5520124Z Receiving objects: 26% (7592/29198) 2024-05-19T18:41:21.5533306Z Receiving objects: 27% (7884/29198) 2024-05-19T18:41:21.5552391Z Receiving objects: 28% (8176/29198) 2024-05-19T18:41:21.6219407Z Receiving objects: 29% (8468/29198) 2024-05-19T18:41:21.6231954Z Receiving objects: 30% (8760/29198) 2024-05-19T18:41:21.6264161Z Receiving objects: 31% (9052/29198) 2024-05-19T18:41:21.8071962Z Receiving objects: 32% (9344/29198) 2024-05-19T18:41:22.4215027Z Receiving objects: 33% (9636/29198) 2024-05-19T18:41:22.6709188Z Receiving objects: 33% (9774/29198), 73.91 MiB | 73.91 MiB/s 2024-05-19T18:41:23.1601069Z Receiving objects: 34% (9928/29198), 73.91 MiB | 73.91 MiB/s 2024-05-19T18:41:23.2260000Z Receiving objects: 35% (10220/29198), 111.77 MiB | 74.56 MiB/s 2024-05-19T18:41:23.2923263Z Receiving objects: 36% (10512/29198), 111.77 MiB | 74.56 MiB/s 2024-05-19T18:41:23.3581985Z Receiving objects: 37% (10804/29198), 111.77 MiB | 74.56 MiB/s 2024-05-19T18:41:23.4213444Z Receiving objects: 38% (11096/29198), 111.77 MiB | 74.56 MiB/s 2024-05-19T18:41:23.4237399Z Receiving objects: 38% (11376/29198), 150.25 MiB | 75.16 MiB/s 2024-05-19T18:41:23.4896473Z Receiving objects: 39% (11388/29198), 150.25 MiB | 75.16 MiB/s 2024-05-19T18:41:23.5561114Z Receiving objects: 40% (11680/29198), 150.25 MiB | 75.16 MiB/s 2024-05-19T18:41:23.6220081Z Receiving objects: 41% (11972/29198), 150.25 MiB | 75.16 MiB/s 2024-05-19T18:41:23.6837166Z Receiving objects: 42% (12264/29198), 150.25 MiB | 75.16 MiB/s 2024-05-19T18:41:23.9521498Z Receiving objects: 43% (12556/29198), 150.25 MiB | 75.16 MiB/s 2024-05-19T18:41:23.9655137Z Receiving objects: 44% (12848/29198), 185.48 MiB | 74.22 MiB/s 2024-05-19T18:41:24.0001162Z Receiving objects: 45% (13140/29198), 185.48 MiB | 74.22 MiB/s 2024-05-19T18:41:24.1630066Z Receiving objects: 46% (13432/29198), 185.48 MiB | 74.22 MiB/s 2024-05-19T18:41:24.4215369Z Receiving objects: 47% (13724/29198), 185.48 MiB | 74.22 MiB/s 2024-05-19T18:41:24.4263711Z Receiving objects: 47% (13886/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4650854Z Receiving objects: 48% (14016/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4660253Z Receiving objects: 49% (14308/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4666618Z Receiving objects: 50% (14599/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4682393Z Receiving objects: 51% (14891/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4689995Z Receiving objects: 52% (15183/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4821690Z Receiving objects: 53% (15475/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4827226Z Receiving objects: 54% (15767/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4832138Z Receiving objects: 55% (16059/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4838148Z Receiving objects: 56% (16351/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4843737Z Receiving objects: 57% (16643/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4854282Z Receiving objects: 58% (16935/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.4862876Z Receiving objects: 59% (17227/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5257060Z Receiving objects: 60% (17519/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5264163Z Receiving objects: 61% (17811/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5276477Z Receiving objects: 62% (18103/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5286708Z Receiving objects: 63% (18395/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5299561Z Receiving objects: 64% (18687/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5307852Z Receiving objects: 65% (18979/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5326914Z Receiving objects: 66% (19271/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5331904Z Receiving objects: 67% (19563/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5339179Z Receiving objects: 68% (19855/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.5345460Z Receiving objects: 69% (20147/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7344694Z Receiving objects: 70% (20439/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7556516Z Receiving objects: 71% (20731/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7562846Z Receiving objects: 72% (21023/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7570634Z Receiving objects: 73% (21315/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7581292Z Receiving objects: 74% (21607/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7586065Z Receiving objects: 75% (21899/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7719277Z Receiving objects: 76% (22191/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7724848Z Receiving objects: 77% (22483/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7731855Z Receiving objects: 78% (22775/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7737862Z Receiving objects: 79% (23067/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7742410Z Receiving objects: 80% (23359/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7748241Z Receiving objects: 81% (23651/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7859881Z Receiving objects: 82% (23943/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7876399Z Receiving objects: 83% (24235/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7886356Z Receiving objects: 84% (24527/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7906171Z Receiving objects: 85% (24819/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7918219Z Receiving objects: 86% (25111/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7922403Z Receiving objects: 87% (25403/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7925985Z Receiving objects: 88% (25695/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7928456Z Receiving objects: 89% (25987/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7933715Z Receiving objects: 90% (26279/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7936572Z Receiving objects: 91% (26571/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7939923Z Receiving objects: 92% (26863/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.7945241Z Receiving objects: 93% (27155/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8005637Z Receiving objects: 94% (27447/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8064592Z Receiving objects: 95% (27739/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8100347Z Receiving objects: 96% (28031/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8116717Z Receiving objects: 97% (28323/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8131575Z Receiving objects: 98% (28615/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8160807Z Receiving objects: 99% (28907/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8161868Z remote: Total 29198 (delta 3632), reused 3517 (delta 3288), pack-reused 25077 2024-05-19T18:41:24.8177979Z Receiving objects: 100% (29198/29198), 221.61 MiB | 73.89 MiB/s 2024-05-19T18:41:24.8178761Z Receiving objects: 100% (29198/29198), 244.92 MiB | 72.11 MiB/s, done. 2024-05-19T18:41:24.8206042Z Resolving deltas: 0% (0/15773) 2024-05-19T18:41:24.8231410Z Resolving deltas: 1% (161/15773) 2024-05-19T18:41:24.8254805Z Resolving deltas: 2% (350/15773) 2024-05-19T18:41:24.8293337Z Resolving deltas: 3% (491/15773) 2024-05-19T18:41:24.8307267Z Resolving deltas: 4% (641/15773) 2024-05-19T18:41:24.8314266Z Resolving deltas: 5% (871/15773) 2024-05-19T18:41:24.8317936Z Resolving deltas: 6% (1048/15773) 2024-05-19T18:41:24.8345796Z Resolving deltas: 7% (1108/15773) 2024-05-19T18:41:24.8379759Z Resolving deltas: 8% (1266/15773) 2024-05-19T18:41:24.8411934Z Resolving deltas: 9% (1421/15773) 2024-05-19T18:41:24.8428128Z Resolving deltas: 10% (1579/15773) 2024-05-19T18:41:24.8465926Z Resolving deltas: 11% (1749/15773) 2024-05-19T18:41:24.8484948Z Resolving deltas: 12% (1893/15773) 2024-05-19T18:41:24.8500458Z Resolving deltas: 13% (2051/15773) 2024-05-19T18:41:24.8529696Z Resolving deltas: 14% (2209/15773) 2024-05-19T18:41:24.8542894Z Resolving deltas: 15% (2380/15773) 2024-05-19T18:41:24.8550560Z Resolving deltas: 16% (2531/15773) 2024-05-19T18:41:24.8560747Z Resolving deltas: 17% (2682/15773) 2024-05-19T18:41:24.8569698Z Resolving deltas: 18% (2858/15773) 2024-05-19T18:41:24.8585392Z Resolving deltas: 19% (3000/15773) 2024-05-19T18:41:24.8599450Z Resolving deltas: 20% (3157/15773) 2024-05-19T18:41:24.8615910Z Resolving deltas: 21% (3314/15773) 2024-05-19T18:41:24.8629212Z Resolving deltas: 22% (3482/15773) 2024-05-19T18:41:24.8643204Z Resolving deltas: 23% (3630/15773) 2024-05-19T18:41:24.8651717Z Resolving deltas: 24% (3787/15773) 2024-05-19T18:41:24.8665320Z Resolving deltas: 25% (3944/15773) 2024-05-19T18:41:24.8680081Z Resolving deltas: 26% (4102/15773) 2024-05-19T18:41:24.8704631Z Resolving deltas: 27% (4273/15773) 2024-05-19T18:41:24.8721785Z Resolving deltas: 28% (4417/15773) 2024-05-19T18:41:24.8732593Z Resolving deltas: 29% (4580/15773) 2024-05-19T18:41:24.8741053Z Resolving deltas: 30% (4732/15773) 2024-05-19T18:41:24.8756354Z Resolving deltas: 31% (4910/15773) 2024-05-19T18:41:24.8778787Z Resolving deltas: 32% (5180/15773) 2024-05-19T18:41:24.8789896Z Resolving deltas: 33% (5251/15773) 2024-05-19T18:41:24.8797953Z Resolving deltas: 34% (5369/15773) 2024-05-19T18:41:24.8807595Z Resolving deltas: 35% (5525/15773) 2024-05-19T18:41:24.8814993Z Resolving deltas: 36% (5682/15773) 2024-05-19T18:41:24.8822872Z Resolving deltas: 37% (5859/15773) 2024-05-19T18:41:24.8828731Z Resolving deltas: 38% (5997/15773) 2024-05-19T18:41:24.8834981Z Resolving deltas: 39% (6159/15773) 2024-05-19T18:41:24.8843177Z Resolving deltas: 40% (6311/15773) 2024-05-19T18:41:24.8849389Z Resolving deltas: 41% (6502/15773) 2024-05-19T18:41:24.8856528Z Resolving deltas: 42% (6652/15773) 2024-05-19T18:41:24.8870399Z Resolving deltas: 43% (6811/15773) 2024-05-19T18:41:24.8875021Z Resolving deltas: 44% (7040/15773) 2024-05-19T18:41:24.8881381Z Resolving deltas: 45% (7112/15773) 2024-05-19T18:41:24.8887346Z Resolving deltas: 46% (7265/15773) 2024-05-19T18:41:24.8894775Z Resolving deltas: 47% (7416/15773) 2024-05-19T18:41:24.8907721Z Resolving deltas: 48% (7587/15773) 2024-05-19T18:41:24.8918956Z Resolving deltas: 49% (7732/15773) 2024-05-19T18:41:24.8926511Z Resolving deltas: 50% (7889/15773) 2024-05-19T18:41:24.8936125Z Resolving deltas: 51% (8066/15773) 2024-05-19T18:41:24.8949826Z Resolving deltas: 52% (8219/15773) 2024-05-19T18:41:24.8960530Z Resolving deltas: 53% (8361/15773) 2024-05-19T18:41:24.8967823Z Resolving deltas: 54% (8518/15773) 2024-05-19T18:41:24.8973362Z Resolving deltas: 55% (8694/15773) 2024-05-19T18:41:24.8979943Z Resolving deltas: 56% (8834/15773) 2024-05-19T18:41:24.8985536Z Resolving deltas: 57% (8993/15773) 2024-05-19T18:41:24.8990534Z Resolving deltas: 58% (9151/15773) 2024-05-19T18:41:24.9001937Z Resolving deltas: 59% (9307/15773) 2024-05-19T18:41:24.9007457Z Resolving deltas: 60% (9525/15773) 2024-05-19T18:41:24.9012184Z Resolving deltas: 61% (9645/15773) 2024-05-19T18:41:24.9025773Z Resolving deltas: 62% (9781/15773) 2024-05-19T18:41:24.9028759Z Resolving deltas: 63% (10052/15773) 2024-05-19T18:41:24.9035135Z Resolving deltas: 64% (10115/15773) 2024-05-19T18:41:24.9043680Z Resolving deltas: 65% (10257/15773) 2024-05-19T18:41:24.9051313Z Resolving deltas: 66% (10416/15773) 2024-05-19T18:41:24.9059704Z Resolving deltas: 67% (10571/15773) 2024-05-19T18:41:24.9073023Z Resolving deltas: 68% (10743/15773) 2024-05-19T18:41:24.9080514Z Resolving deltas: 69% (10890/15773) 2024-05-19T18:41:24.9088567Z Resolving deltas: 70% (11056/15773) 2024-05-19T18:41:24.9093765Z Resolving deltas: 71% (11224/15773) 2024-05-19T18:41:24.9101119Z Resolving deltas: 72% (11371/15773) 2024-05-19T18:41:24.9107603Z Resolving deltas: 73% (11515/15773) 2024-05-19T18:41:24.9114231Z Resolving deltas: 74% (11678/15773) 2024-05-19T18:41:24.9119409Z Resolving deltas: 75% (11844/15773) 2024-05-19T18:41:24.9125117Z Resolving deltas: 76% (11990/15773) 2024-05-19T18:41:24.9135273Z Resolving deltas: 77% (12146/15773) 2024-05-19T18:41:24.9141712Z Resolving deltas: 78% (12363/15773) 2024-05-19T18:41:24.9163182Z Resolving deltas: 79% (12493/15773) 2024-05-19T18:41:24.9166864Z Resolving deltas: 81% (12826/15773) 2024-05-19T18:41:24.9174023Z Resolving deltas: 82% (12936/15773) 2024-05-19T18:41:24.9180107Z Resolving deltas: 83% (13097/15773) 2024-05-19T18:41:24.9189894Z Resolving deltas: 84% (13252/15773) 2024-05-19T18:41:24.9196137Z Resolving deltas: 85% (13447/15773) 2024-05-19T18:41:24.9202529Z Resolving deltas: 86% (13580/15773) 2024-05-19T18:41:24.9208892Z Resolving deltas: 87% (13724/15773) 2024-05-19T18:41:24.9219699Z Resolving deltas: 88% (13892/15773) 2024-05-19T18:41:24.9226795Z Resolving deltas: 89% (14064/15773) 2024-05-19T18:41:24.9240946Z Resolving deltas: 90% (14230/15773) 2024-05-19T18:41:24.9258229Z Resolving deltas: 91% (14393/15773) 2024-05-19T18:41:24.9267040Z Resolving deltas: 92% (14550/15773) 2024-05-19T18:41:24.9280000Z Resolving deltas: 93% (14681/15773) 2024-05-19T18:41:24.9290210Z Resolving deltas: 94% (14840/15773) 2024-05-19T18:41:24.9350216Z Resolving deltas: 95% (14990/15773) 2024-05-19T18:41:24.9352705Z Resolving deltas: 96% (15278/15773) 2024-05-19T18:41:24.9372615Z Resolving deltas: 97% (15308/15773) 2024-05-19T18:41:24.9385034Z Resolving deltas: 98% (15482/15773) 2024-05-19T18:41:24.9405821Z Resolving deltas: 99% (15616/15773) 2024-05-19T18:41:24.9406607Z Resolving deltas: 100% (15773/15773) 2024-05-19T18:41:24.9407051Z Resolving deltas: 100% (15773/15773), done. 2024-05-19T18:41:25.9930894Z + pushd torchbench 2024-05-19T18:41:25.9931330Z ~/workspace/torchbench ~/workspace 2024-05-19T18:41:25.9931921Z + git checkout d6015d42d9a1834bc7595c4bd6852562fb80b30b 2024-05-19T18:41:26.0433161Z Note: switching to 'd6015d42d9a1834bc7595c4bd6852562fb80b30b'. 2024-05-19T18:41:26.0433749Z 2024-05-19T18:41:26.0434288Z You are in 'detached HEAD' state. You can look around, make experimental 2024-05-19T18:41:26.0435386Z changes and commit them, and you can discard any commits you make in this 2024-05-19T18:41:26.0436444Z state without impacting any branches by switching back to a branch. 2024-05-19T18:41:26.0436889Z 2024-05-19T18:41:26.0437207Z If you want to create a new branch to retain commits you create, you may 2024-05-19T18:41:26.0438269Z do so (now or later) by using -c with the switch command. Example: 2024-05-19T18:41:26.0438717Z 2024-05-19T18:41:26.0438903Z git switch -c 2024-05-19T18:41:26.0439178Z 2024-05-19T18:41:26.0439330Z Or undo this operation with: 2024-05-19T18:41:26.0439580Z 2024-05-19T18:41:26.0439716Z git switch - 2024-05-19T18:41:26.0439899Z 2024-05-19T18:41:26.0440236Z Turn off this advice by setting config variable advice.detachedHead to false 2024-05-19T18:41:26.0440728Z 2024-05-19T18:41:26.0440949Z HEAD is now at d6015d42 Add sam_fast torchbench (#2182) 2024-05-19T18:41:26.0441463Z + '[' '' ']' 2024-05-19T18:41:26.0441836Z + python install.py --continue_on_fail 2024-05-19T18:41:27.8479194Z checking packages torch, torchvision, torchaudio are installed...OK 2024-05-19T18:41:59.1476837Z Failed to install torchbenchmark requirements: 2024-05-19T18:41:59.1484209Z b" error: subprocess-exited-with-error\n \n \xc3\x97 git clone --filter=blob:none --quiet https://github.com/huggingface/pytorch-image-models.git /tmp/pip-req-build-xaf0g4ay did not run successfully.\n \xe2\x94\x82 exit code: 128\n \xe2\x95\xb0\xe2\x94\x80> [7 lines of output]\n fatal: the remote end hung up unexpectedly\n fatal: early EOF\n fatal: index-pack failed\n warning: Clone succeeded, but checkout failed.\n You can inspect what was checked out with 'git status'\n and retry with 'git restore --source=HEAD :/'\n \n [end of output]\n \n note: This error originates from a subprocess, and is likely not a problem with pip.\nerror: subprocess-exited-with-error\n\n\xc3\x97 git clone --filter=blob:none --quiet https://github.com/huggingface/pytorch-image-models.git /tmp/pip-req-build-xaf0g4ay did not run successfully.\n\xe2\x94\x82 exit code: 128\n\xe2\x95\xb0\xe2\x94\x80> See above for output.\n\nnote: This error originates from a subprocess, and is likely not a problem with pip.\n" 2024-05-19T18:42:01.3588886Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-05-19T18:42:03.5959256Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-05-19T18:42:03.6107076Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch...FAIL 2024-05-19T18:42:06.1624630Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-05-19T18:42:08.3718009Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-05-19T18:42:08.3868220Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-05-19T18:42:16.4837341Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-05-19T18:42:19.6061967Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-05-19T18:42:22.7463262Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-05-19T18:42:25.8702696Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-05-19T18:42:25.8705825Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-05-19T18:42:27.5951419Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-05-19T18:42:30.9871821Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-05-19T18:42:31.0017686Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-05-19T18:43:25.5952571Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-05-19T18:43:35.2637540Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-05-19T18:43:42.6499149Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-05-19T18:43:49.6903654Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-05-19T18:44:45.8299796Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...FAIL 2024-05-19T18:44:53.0272594Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-05-19T18:44:59.5186617Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-05-19T18:45:06.7810832Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-05-19T18:45:14.2056870Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-05-19T18:45:21.7784103Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-05-19T18:45:28.8646997Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-05-19T18:45:36.0128479Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-05-19T18:45:39.9131954Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-05-19T18:46:23.3191471Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-05-19T18:46:45.6773563Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-05-19T18:46:54.2984833Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-05-19T18:46:54.3187805Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...FAIL 2024-05-19T18:46:56.3391865Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-05-19T18:46:58.6786248Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-05-19T18:47:05.6327161Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-05-19T18:47:12.8073400Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-05-19T18:47:18.6859574Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-05-19T18:47:26.4628985Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-05-19T18:47:32.6507006Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-05-19T18:47:38.1037997Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-05-19T18:47:44.5131678Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-05-19T18:48:00.4261957Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-05-19T18:48:06.8912023Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-05-19T18:48:11.3065257Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-05-19T18:48:16.4470029Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-05-19T18:48:23.3792713Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-05-19T18:48:23.3795873Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-05-19T18:48:35.9165039Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-05-19T18:48:42.3244638Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-05-19T18:48:42.3246555Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-05-19T18:48:49.9499335Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-05-19T18:48:52.0735806Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-05-19T18:48:54.2251738Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-05-19T18:49:33.5217266Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-05-19T18:50:53.2482394Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-05-19T18:50:53.2640513Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-05-19T18:50:55.6540117Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-05-19T18:50:55.6697659Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-05-19T18:50:55.6857272Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-05-19T18:50:55.7012572Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-05-19T18:50:55.7167096Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-05-19T18:50:55.7340402Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-05-19T18:51:15.2841737Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-05-19T18:51:15.2843255Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-05-19T18:51:17.4550132Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-05-19T18:51:20.6706140Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-05-19T18:51:20.6863555Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-05-19T18:51:20.7026440Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-05-19T18:51:20.7179435Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-05-19T18:51:20.7331950Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-05-19T18:51:20.7485100Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-05-19T18:51:27.5899047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-05-19T18:51:30.0173299Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-05-19T18:51:36.3107820Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-05-19T18:51:36.3267773Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-05-19T18:51:36.3426771Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-05-19T18:51:36.3585082Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-05-19T18:51:36.3740348Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-05-19T18:51:36.3895783Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-05-19T18:51:48.9981562Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-05-19T18:52:03.2196071Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-05-19T18:52:03.2359511Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-05-19T18:52:03.2362792Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-05-19T18:52:03.2366175Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-05-19T18:52:06.7664114Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-05-19T18:52:09.5003547Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-05-19T18:52:09.5158685Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-05-19T18:52:36.2001361Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-05-19T18:52:47.8375065Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-05-19T18:52:53.8139908Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-05-19T18:52:53.8306122Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...FAIL 2024-05-19T18:52:53.8467449Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-05-19T18:52:53.8624995Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-05-19T18:52:53.8783943Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-05-19T18:52:53.8941359Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-05-19T18:52:53.9098808Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-05-19T18:52:53.9255557Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-05-19T18:52:53.9411765Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-05-19T18:53:08.0759661Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-05-19T18:53:21.2995131Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-05-19T18:53:21.3150125Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-05-19T18:53:26.3662365Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-05-19T18:53:29.4074438Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-05-19T18:53:29.4075496Z Error for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch: 2024-05-19T18:53:29.4078621Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4079373Z Traceback (most recent call last): 2024-05-19T18:53:29.4080988Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch/install.py", line 2, in 2024-05-19T18:53:29.4081861Z import patch 2024-05-19T18:53:29.4082304Z ModuleNotFoundError: No module named 'patch' 2024-05-19T18:53:29.4082642Z 2024-05-19T18:53:29.4082665Z 2024-05-19T18:53:29.4082670Z 2024-05-19T18:53:29.4082938Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4083302Z 2024-05-19T18:53:29.4083921Z Error for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5: 2024-05-19T18:53:29.4085197Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4086529Z decompressing input tarball: /var/lib/jenkins/workspace/torchbench/torchbenchmark/data/coco2017-minimal.tar.gz... error: subprocess-exited-with-error 2024-05-19T18:53:29.4087588Z 2024-05-19T18:53:29.4088561Z × git clone --filter=blob:none --quiet https://github.com/facebookresearch/detectron2.git /tmp/pip-req-build-or28fovc did not run successfully. 2024-05-19T18:53:29.4089677Z │ exit code: 128 2024-05-19T18:53:29.4090055Z ╰─> [4 lines of output] 2024-05-19T18:53:29.4090645Z error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:53:29.4091355Z fatal: the remote end hung up unexpectedly 2024-05-19T18:53:29.4091823Z fatal: early EOF 2024-05-19T18:53:29.4092209Z fatal: index-pack failed 2024-05-19T18:53:29.4092598Z [end of output] 2024-05-19T18:53:29.4092943Z 2024-05-19T18:53:29.4093462Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:53:29.4094174Z error: subprocess-exited-with-error 2024-05-19T18:53:29.4094474Z 2024-05-19T18:53:29.4095265Z × git clone --filter=blob:none --quiet https://github.com/facebookresearch/detectron2.git /tmp/pip-req-build-or28fovc did not run successfully. 2024-05-19T18:53:29.4096478Z │ exit code: 128 2024-05-19T18:53:29.4097011Z ╰─> See above for output. 2024-05-19T18:53:29.4097352Z 2024-05-19T18:53:29.4097848Z note: This error originates from a subprocess, and is likely not a problem with pip. 2024-05-19T18:53:29.4098474Z OK 2024-05-19T18:53:29.4098762Z Traceback (most recent call last): 2024-05-19T18:53:29.4099731Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5/install.py", line 7, in 2024-05-19T18:53:29.4100718Z install_detectron2(MODEL_NAME, MODEL_DIR) 2024-05-19T18:53:29.4101735Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/__init__.py", line 59, in install_detectron2 2024-05-19T18:53:29.4102726Z pip_install_requirements() 2024-05-19T18:53:29.4103713Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/__init__.py", line 41, in pip_install_requirements 2024-05-19T18:53:29.4105236Z subprocess.check_call([sys.executable, '-m', 'pip', 'install', '-q', '-r', requirements_file]) 2024-05-19T18:53:29.4106179Z File "/opt/conda/envs/py_3.10/lib/python3.10/subprocess.py", line 369, in check_call 2024-05-19T18:53:29.4106876Z raise CalledProcessError(retcode, cmd) 2024-05-19T18:53:29.4108617Z subprocess.CalledProcessError: Command '['/opt/conda/envs/py_3.10/bin/python', '-m', 'pip', 'install', '-q', '-r', '/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/requirements.txt']' returned non-zero exit status 1. 2024-05-19T18:53:29.4110076Z 2024-05-19T18:53:29.4110085Z 2024-05-19T18:53:29.4110092Z 2024-05-19T18:53:29.4110371Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4110746Z 2024-05-19T18:53:29.4111324Z Error for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert: 2024-05-19T18:53:29.4112169Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4112747Z Traceback (most recent call last): 2024-05-19T18:53:29.4113623Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert/install.py", line 4, in 2024-05-19T18:53:29.4114621Z import patch 2024-05-19T18:53:29.4115047Z ModuleNotFoundError: No module named 'patch' 2024-05-19T18:53:29.4115383Z 2024-05-19T18:53:29.4115388Z 2024-05-19T18:53:29.4115394Z 2024-05-19T18:53:29.4115661Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4116038Z 2024-05-19T18:53:29.4116469Z Error for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet: 2024-05-19T18:53:29.4117336Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4118158Z Traceback (most recent call last): 2024-05-19T18:53:29.4119092Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet/install.py", line 3, in 2024-05-19T18:53:29.4119972Z import patch 2024-05-19T18:53:29.4120414Z ModuleNotFoundError: No module named 'patch' 2024-05-19T18:53:29.4120767Z 2024-05-19T18:53:29.4120783Z 2024-05-19T18:53:29.4120788Z 2024-05-19T18:53:29.4121059Z --------------------------------------------------------------------------- 2024-05-19T18:53:29.4121428Z 2024-05-19T18:53:29.4121677Z Warning: some benchmarks were not installed due to failure 2024-05-19T18:53:29.8164927Z + popd 2024-05-19T18:53:29.8165276Z ~/workspace 2024-05-19T18:53:29.8165682Z + [[ inductor_torchbench != *cpu_inductor* ]] 2024-05-19T18:53:29.8166165Z + install_torchrec_and_fbgemm 2024-05-19T18:53:29.8166543Z + local torchrec_commit 2024-05-19T18:53:29.8168756Z ++ get_pinned_commit torchrec 2024-05-19T18:53:29.8169203Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-05-19T18:53:29.8221192Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T18:53:29.8221713Z + local fbgemm_commit 2024-05-19T18:53:29.8223401Z ++ get_pinned_commit fbgemm 2024-05-19T18:53:29.8223826Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-05-19T18:53:29.8232753Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:53:29.8233368Z + pip_uninstall torchrec-nightly 2024-05-19T18:53:29.8233814Z + pip uninstall -y torchrec-nightly 2024-05-19T18:53:30.1841629Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-05-19T18:53:30.2292086Z + pip_uninstall fbgemm-gpu-nightly 2024-05-19T18:53:30.2292613Z + pip uninstall -y fbgemm-gpu-nightly 2024-05-19T18:53:30.5681614Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-05-19T18:53:30.6109295Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-05-19T18:53:30.6110255Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-05-19T18:53:31.0578846Z Collecting setuptools-git-versioning 2024-05-19T18:53:31.0753281Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-05-19T18:53:31.1567208Z Collecting scikit-build 2024-05-19T18:53:31.1614953Z Downloading scikit_build-0.17.6-py3-none-any.whl.metadata (14 kB) 2024-05-19T18:53:31.2312635Z Collecting pyre-extensions 2024-05-19T18:53:31.2353644Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-05-19T18:53:31.2823802Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.0) 2024-05-19T18:53:31.2827678Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-05-19T18:53:31.3345526Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-05-19T18:53:31.3378466Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-05-19T18:53:31.4554621Z Collecting distro (from scikit-build) 2024-05-19T18:53:31.4586635Z Downloading distro-1.9.0-py3-none-any.whl.metadata (6.8 kB) 2024-05-19T18:53:31.5041226Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-05-19T18:53:31.5046488Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-05-19T18:53:31.5594264Z Collecting typing-inspect (from pyre-extensions) 2024-05-19T18:53:31.5632439Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-05-19T18:53:31.6077960Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.11.0) 2024-05-19T18:53:31.6309535Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-05-19T18:53:31.6509739Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-05-19T18:53:31.6988527Z Downloading scikit_build-0.17.6-py3-none-any.whl (84 kB) 2024-05-19T18:53:31.7487343Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-05-19T18:53:31.7954516Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-05-19T18:53:31.8444225Z Downloading distro-1.9.0-py3-none-any.whl (20 kB) 2024-05-19T18:53:31.8919084Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-05-19T18:53:33.8901941Z Installing collected packages: typing-inspect, toml, distro, setuptools-git-versioning, scikit-build, pyre-extensions 2024-05-19T18:53:34.2867641Z Successfully installed distro-1.9.0 pyre-extensions-0.0.30 scikit-build-0.17.6 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-05-19T18:53:34.4032882Z + CUDA_PATH=/usr/local/cuda-12.1 2024-05-19T18:53:34.4034192Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-05-19T18:53:34.4036093Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-05-19T18:53:34.7334992Z Collecting fbgemm-gpu 2024-05-19T18:53:34.7338997Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-17j6kjlo/fbgemm-gpu_641339a16d3a47a6b3da98bd816749d6 2024-05-19T18:53:34.7355698Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-17j6kjlo/fbgemm-gpu_641339a16d3a47a6b3da98bd816749d6 2024-05-19T18:53:35.6089654Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-05-19T18:53:35.6108004Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:53:35.9934677Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:53:36.4070546Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-05-19T18:53:36.4072978Z Running command git submodule update --init --recursive -q 2024-05-19T18:54:20.3407591Z error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8) 2024-05-19T18:54:20.3408358Z fatal: the remote end hung up unexpectedly 2024-05-19T18:54:20.3409157Z fatal: early EOF 2024-05-19T18:54:20.3409724Z fatal: index-pack failed 2024-05-19T18:54:20.3432926Z fatal: clone of 'https://github.com/google/googletest' into submodule path '/tmp/pip-install-17j6kjlo/fbgemm-gpu_641339a16d3a47a6b3da98bd816749d6/third_party/googletest' failed 2024-05-19T18:54:20.3434257Z Failed to clone 'third_party/googletest'. Retry scheduled 2024-05-19T18:54:24.1513458Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T18:54:24.1541357Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.26.4) 2024-05-19T18:54:24.1559220Z Building wheels for collected packages: fbgemm-gpu 2024-05-19T19:36:26.7548361Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-05-19T19:36:26.9365706Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253270644 sha256=5e7c6dc6ee41e24e07c6b79b6b84f944b242badb0233c6a9c39fd6090036bfc5 2024-05-19T19:36:26.9367393Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-05-19T19:36:26.9385527Z Successfully built fbgemm-gpu 2024-05-19T19:36:28.9008596Z Installing collected packages: fbgemm-gpu 2024-05-19T19:36:32.0451686Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-05-19T19:36:32.6603815Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:36:32.6605370Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:36:32.9944501Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:36:32.9948377Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-llpx0khx 2024-05-19T19:36:32.9964578Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-llpx0khx 2024-05-19T19:36:33.6726295Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-05-19T19:36:33.6741143Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:36:33.9432879Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:36:34.2722103Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-05-19T19:36:34.4954811Z Preparing metadata (setup.py) ... [?25l- done 2024-05-19T19:36:34.5011356Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-05-19T19:36:34.5014143Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-05-19T19:36:34.5018256Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-05-19T19:36:34.5526499Z Collecting torchmetrics (from torchrec==0.3.2) 2024-05-19T19:36:34.5706746Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-05-19T19:36:34.5764154Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-05-19T19:36:34.5767387Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-05-19T19:36:34.5781089Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.26.4) 2024-05-19T19:36:34.5802492Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.8.2) 2024-05-19T19:36:34.6467823Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-05-19T19:36:34.6472101Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-05-19T19:36:34.6475851Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-05-19T19:36:34.7253505Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.0) 2024-05-19T19:36:34.7258072Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.4.0a0+gitc8c7ddf) 2024-05-19T19:36:34.7446348Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-05-19T19:36:34.7483691Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-05-19T19:36:34.7654404Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-05-19T19:36:34.7657376Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.11.0) 2024-05-19T19:36:34.7700956Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-05-19T19:36:34.7770671Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-05-19T19:36:34.7774295Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12) 2024-05-19T19:36:34.7777481Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-05-19T19:36:34.7780735Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-05-19T19:36:34.7784223Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-05-19T19:36:34.8448306Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-05-19T19:36:34.8751380Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-05-19T19:36:34.8925317Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-05-19T19:36:34.9135909Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-05-19T19:36:34.9233979Z Building wheels for collected packages: torchrec 2024-05-19T19:36:35.3383510Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-05-19T19:36:35.3393281Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=c50c823b25637b11b413924ce356c0842eb783603d37bc941271a0b337020617 2024-05-19T19:36:35.3395037Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-05-19T19:36:35.3415787Z Successfully built torchrec 2024-05-19T19:36:37.2113802Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-05-19T19:36:37.7614684Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-05-19T19:36:37.8945090Z ++ pwd 2024-05-19T19:36:37.8945928Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-05-19T19:36:37.8946465Z + test_dynamo_benchmark torchbench 0 2024-05-19T19:36:37.8949182Z ++ pwd 2024-05-19T19:36:37.8951736Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:36:37.8952301Z + local suite=torchbench 2024-05-19T19:36:37.8952680Z + shift 2024-05-19T19:36:37.8953046Z + local shard_id=0 2024-05-19T19:36:37.8953359Z + shift 2024-05-19T19:36:37.8954785Z + [[ inductor_torchbench == *perf_compare* ]] 2024-05-19T19:36:37.8955400Z + [[ inductor_torchbench == *perf* ]] 2024-05-19T19:36:37.8956001Z + [[ inductor_torchbench == *cpu_inductor* ]] 2024-05-19T19:36:37.8956724Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-05-19T19:36:37.8957995Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-05-19T19:36:37.8958827Z ++ pwd 2024-05-19T19:36:37.8959372Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:36:37.8960042Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-05-19T19:36:37.8986949Z + local name=inference 2024-05-19T19:36:37.8987482Z + shift 2024-05-19T19:36:37.8987869Z + local suite=torchbench 2024-05-19T19:36:37.8988344Z + shift 2024-05-19T19:36:37.8988719Z + local shard_id=0 2024-05-19T19:36:37.8989124Z + shift 2024-05-19T19:36:37.8989401Z + partition_flags=() 2024-05-19T19:36:37.8989742Z + local partition_flags 2024-05-19T19:36:37.8990130Z + [[ -n 2 ]] 2024-05-19T19:36:37.8990440Z + [[ -n 0 ]] 2024-05-19T19:36:37.8991169Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-05-19T19:36:37.8991867Z + [[ inductor_torchbench == *perf_compare* ]] 2024-05-19T19:36:37.8992336Z + [[ inductor_torchbench == *perf* ]] 2024-05-19T19:36:37.8992787Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-05-19T19:36:37.8994377Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-05-19T19:36:42.9556430Z 2024-05-19T19:36:43.9404889Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:36:43.9405516Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:36:43.9405988Z cuda eval torchrec_dlrm 2024-05-19T19:36:43.9414204Z Traceback (most recent call last): 2024-05-19T19:36:43.9415286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T19:36:43.9416569Z self.model_iter_fn(model, example_inputs) 2024-05-19T19:36:43.9417371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 426, in forward_pass 2024-05-19T19:36:43.9418097Z return mod(*inputs) 2024-05-19T19:36:43.9419125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:36:43.9420026Z return self._call_impl(*args, **kwargs) 2024-05-19T19:36:43.9420968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:36:43.9421815Z return forward_call(*args, **kwargs) 2024-05-19T19:36:43.9422701Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-05-19T19:36:43.9423628Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-05-19T19:36:43.9424785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:36:43.9425681Z return self._call_impl(*args, **kwargs) 2024-05-19T19:36:43.9426720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:36:43.9427562Z return forward_call(*args, **kwargs) 2024-05-19T19:36:43.9428427Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-05-19T19:36:43.9429234Z concatenated_dense = self.inter_arch( 2024-05-19T19:36:43.9430213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:36:43.9431243Z return self._call_impl(*args, **kwargs) 2024-05-19T19:36:43.9435716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:36:43.9436901Z return forward_call(*args, **kwargs) 2024-05-19T19:36:43.9438278Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-05-19T19:36:43.9439232Z return self.crossnet(combined_values.reshape([B, -1])) 2024-05-19T19:36:43.9440282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:36:43.9441169Z return self._call_impl(*args, **kwargs) 2024-05-19T19:36:43.9442236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:36:43.9443064Z return forward_call(*args, **kwargs) 2024-05-19T19:36:43.9443966Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-05-19T19:36:43.9444896Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-05-19T19:36:43.9445731Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-05-19T19:36:43.9446278Z 2024-05-19T19:36:43.9446585Z The above exception was the direct cause of the following exception: 2024-05-19T19:36:43.9447033Z 2024-05-19T19:36:43.9447177Z Traceback (most recent call last): 2024-05-19T19:36:43.9447821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:36:43.9448471Z ) = runner.load_model( 2024-05-19T19:36:43.9449127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-05-19T19:36:43.9449874Z self.validate_model(model, example_inputs) 2024-05-19T19:36:43.9450622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T19:36:43.9451379Z raise RuntimeError("Eager run failed") from e 2024-05-19T19:36:43.9451845Z RuntimeError: Eager run failed 2024-05-19T19:36:43.9452096Z 2024-05-19T19:36:43.9452212Z eager_fail_to_run 2024-05-19T19:36:46.9842462Z 2024-05-19T19:36:48.9870216Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:36:48.9871548Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:36:48.9932094Z cuda eval BERT_pytorch 2024-05-19T19:37:08.2486412Z pass 2024-05-19T19:37:08.2498611Z TIMING: code_gen:4.959 inductor_compile:8.49225 backend_compile:13.82924 entire_frame_compile:17.10708 2024-05-19T19:37:08.2500776Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:3212 | FakeTensorMode.__torch_dispatch__:36087 | ProxyTorchDispatchMode.__torch_dispatch__:12905 2024-05-19T19:37:08.2502566Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique) 2024-05-19T19:37:11.9111905Z 2024-05-19T19:37:14.5116334Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:37:14.5116845Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:37:14.5167031Z cuda eval Background_Matting 2024-05-19T19:37:14.5169919Z pass_due_to_skip 2024-05-19T19:37:14.5710315Z TIMING: 2024-05-19T19:37:14.5711355Z STATS: call_* op count: 0 2024-05-19T19:37:14.5712984Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-05-19T19:37:17.5643152Z 2024-05-19T19:37:17.5733845Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:37:17.5734613Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:37:17.5735069Z cuda eval DALLE2_pytorch 2024-05-19T19:37:17.5739671Z Traceback (most recent call last): 2024-05-19T19:37:17.5740652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:37:17.5741576Z ) = runner.load_model( 2024-05-19T19:37:17.5742407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 237, in load_model 2024-05-19T19:37:17.5743174Z module = importlib.import_module(c) 2024-05-19T19:37:17.5743914Z File "/opt/conda/envs/py_3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module 2024-05-19T19:37:17.5744747Z return _bootstrap._gcd_import(name[level:], package, level) 2024-05-19T19:37:17.5745578Z File "", line 1050, in _gcd_import 2024-05-19T19:37:17.5746518Z File "", line 1027, in _find_and_load 2024-05-19T19:37:17.5747473Z File "", line 1006, in _find_and_load_unlocked 2024-05-19T19:37:17.5748388Z File "", line 688, in _load_unlocked 2024-05-19T19:37:17.5749094Z File "", line 883, in exec_module 2024-05-19T19:37:17.5749851Z File "", line 241, in _call_with_frames_removed 2024-05-19T19:37:17.5750915Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch/__init__.py", line 3, in 2024-05-19T19:37:17.5752285Z from dalle2_pytorch import DALLE2, Unet, Decoder, DiffusionPriorNetwork, DiffusionPrior, OpenAIClipAdapter 2024-05-19T19:37:17.5753334Z ModuleNotFoundError: No module named 'dalle2_pytorch' 2024-05-19T19:37:17.5753700Z 2024-05-19T19:37:17.5753825Z eager_fail_to_run 2024-05-19T19:37:20.3837245Z 2024-05-19T19:37:23.2870371Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:37:23.2870889Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:37:23.2895687Z cuda eval LearningToPaint 2024-05-19T19:37:31.3641840Z pass 2024-05-19T19:37:31.3659429Z TIMING: code_gen:2.07457 inductor_compile:3.17936 backend_compile:5.51648 entire_frame_compile:6.22844 2024-05-19T19:37:31.3660970Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1544 | FakeTensorMode.__torch_dispatch__:11567 | ProxyTorchDispatchMode.__torch_dispatch__:3014 2024-05-19T19:37:31.3662115Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-05-19T19:37:34.9269604Z 2024-05-19T19:37:36.2556125Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-05-19T19:37:36.2725837Z 2024-05-19T19:37:36.2725846Z 2024-05-19T19:37:36.3729944Z 0% 0.00/528M [00:00", line 1050, in _gcd_import 2024-05-19T19:51:02.1895831Z File "", line 1027, in _find_and_load 2024-05-19T19:51:02.1896856Z File "", line 1006, in _find_and_load_unlocked 2024-05-19T19:51:02.1897585Z File "", line 688, in _load_unlocked 2024-05-19T19:51:02.1898296Z File "", line 883, in exec_module 2024-05-19T19:51:02.1899054Z File "", line 241, in _call_with_frames_removed 2024-05-19T19:51:02.1900119Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert/__init__.py", line 12, in 2024-05-19T19:51:02.1901053Z from fastNLP.embeddings import BertEmbedding 2024-05-19T19:51:02.1901744Z ModuleNotFoundError: No module named 'fastNLP' 2024-05-19T19:51:02.1902083Z 2024-05-19T19:51:02.1902208Z eager_fail_to_run 2024-05-19T19:51:05.0118420Z 2024-05-19T19:51:05.8780828Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:51:05.8781383Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:51:05.8797353Z cuda eval functorch_dp_cifar10 2024-05-19T19:51:14.1454447Z pass 2024-05-19T19:51:14.1472329Z TIMING: code_gen:2.68794 inductor_compile:3.8835 backend_compile:6.19473 entire_frame_compile:6.788 2024-05-19T19:51:14.1473701Z STATS: call_* op count: 138 | FakeTensor.__torch_dispatch__:1255 | FakeTensorMode.__torch_dispatch__:7726 | ProxyTorchDispatchMode.__torch_dispatch__:2467 2024-05-19T19:51:14.1474812Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-05-19T19:51:17.6137059Z 2024-05-19T19:51:18.0851331Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:51:18.0851868Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:51:18.0857020Z cuda eval functorch_maml_omniglot 2024-05-19T19:51:22.1874115Z pass 2024-05-19T19:51:22.1885884Z TIMING: code_gen:1.19412 inductor_compile:1.44847 backend_compile:2.55858 entire_frame_compile:2.72597 2024-05-19T19:51:22.1887238Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:226 | FakeTensorMode.__torch_dispatch__:2064 | ProxyTorchDispatchMode.__torch_dispatch__:477 2024-05-19T19:51:22.1888376Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-05-19T19:51:25.4422195Z 2024-05-19T19:51:26.4119120Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:51:26.4119612Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:51:26.4126648Z cuda eval hf_Albert 2024-05-19T19:51:42.2830079Z pass 2024-05-19T19:51:42.3032886Z TIMING: code_gen:2.48238 inductor_compile:4.97195 backend_compile:9.86681 entire_frame_compile:12.25702 2024-05-19T19:51:42.3035480Z STATS: call_* op count: 872 | FakeTensorMode.__torch_dispatch__:31104 | FakeTensor.__torch_dispatch__:3058 | ProxyTorchDispatchMode.__torch_dispatch__:11558 2024-05-19T19:51:42.3038067Z Dynamo produced 1 graphs covering 872 ops with 0 graph breaks (0 unique) 2024-05-19T19:51:45.8942804Z 2024-05-19T19:51:49.4620167Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:51:49.4620989Z loading model: 0it [00:03, ?it/s] 2024-05-19T19:51:49.4682000Z cuda eval hf_Bart 2024-05-19T19:52:11.3445862Z pass 2024-05-19T19:52:11.3715752Z TIMING: code_gen:3.44142 inductor_compile:6.8121 backend_compile:13.43655 entire_frame_compile:17.11524 2024-05-19T19:52:11.3718658Z STATS: call_* op count: 902 | FakeTensorMode.__torch_dispatch__:38730 | FakeTensor.__torch_dispatch__:4639 | ProxyTorchDispatchMode.__torch_dispatch__:13350 2024-05-19T19:52:11.3719849Z Dynamo produced 1 graphs covering 902 ops with 0 graph breaks (0 unique) 2024-05-19T19:52:15.1564826Z 2024-05-19T19:52:17.6151616Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:52:17.6152116Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:52:17.6201244Z cuda eval hf_Bert 2024-05-19T19:52:37.6635206Z pass 2024-05-19T19:52:37.6826274Z TIMING: code_gen:2.16476 inductor_compile:4.67133 backend_compile:11.04144 entire_frame_compile:16.00848 2024-05-19T19:52:37.6828545Z STATS: call_* op count: 734 | FakeTensorMode.__torch_dispatch__:32448 | FakeTensor.__torch_dispatch__:3359 | ProxyTorchDispatchMode.__torch_dispatch__:11836 2024-05-19T19:52:37.6829981Z Dynamo produced 1 graphs covering 734 ops with 0 graph breaks (0 unique) 2024-05-19T19:52:41.3645222Z 2024-05-19T19:52:45.5969932Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:52:45.5970454Z loading model: 0it [00:04, ?it/s] 2024-05-19T19:52:45.6065189Z cuda eval hf_Bert_large 2024-05-19T19:53:23.4209974Z pass 2024-05-19T19:53:23.4413194Z TIMING: code_gen:3.68392 inductor_compile:8.007 backend_compile:20.14403 entire_frame_compile:29.86262 2024-05-19T19:53:23.4415482Z STATS: call_* op count: 1430 | FakeTensorMode.__torch_dispatch__:62904 | FakeTensor.__torch_dispatch__:6515 | ProxyTorchDispatchMode.__torch_dispatch__:23176 2024-05-19T19:53:23.4416661Z Dynamo produced 1 graphs covering 1430 ops with 0 graph breaks (0 unique) 2024-05-19T19:53:27.6254539Z 2024-05-19T19:53:30.4653068Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:53:30.4653625Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:53:30.4703705Z cuda eval hf_BigBird 2024-05-19T19:54:00.6429869Z W0519 19:54:00.642000 140350158811776 torch/_inductor/utils.py:1130] [9/0] DeviceCopy in input program 2024-05-19T19:54:08.0772184Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:54:08.0773940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:54:08.0777685Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:54:08.0778036Z 2024-05-19T19:54:11.7798349Z W0519 19:54:11.779000 140350158811776 torch/_inductor/utils.py:1130] [9/1] DeviceCopy in input program 2024-05-19T19:54:16.2758992Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:54:16.2760798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:54:16.2762052Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:54:16.2762379Z 2024-05-19T19:54:19.2944954Z W0519 19:54:19.293000 140350158811776 torch/_inductor/utils.py:1130] [9/2] DeviceCopy in input program 2024-05-19T19:54:24.0583781Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:54:24.0585708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:54:24.0586905Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:54:24.0587224Z 2024-05-19T19:54:27.0672587Z W0519 19:54:27.066000 140350158811776 torch/_inductor/utils.py:1130] [9/3] DeviceCopy in input program 2024-05-19T19:54:31.8800753Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:54:31.8802533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:54:31.8803746Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:54:31.8804069Z 2024-05-19T19:54:34.8755223Z W0519 19:54:34.874000 140350158811776 torch/_inductor/utils.py:1130] [9/4] DeviceCopy in input program 2024-05-19T19:54:39.7093971Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:54:39.7095722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:54:39.7096934Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:54:39.7097249Z 2024-05-19T19:54:42.7045578Z W0519 19:54:42.704000 140350158811776 torch/_inductor/utils.py:1130] [9/5] DeviceCopy in input program 2024-05-19T19:54:47.2158920Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:54:47.2161015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:54:47.2162220Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:54:47.2162541Z 2024-05-19T19:54:50.5840059Z W0519 19:54:50.583000 140350158811776 torch/_inductor/utils.py:1130] [9/6] DeviceCopy in input program 2024-05-19T19:54:55.0945613Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:54:55.0947440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:54:55.0948719Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:54:55.0949111Z 2024-05-19T19:54:58.4581358Z W0519 19:54:58.457000 140350158811776 torch/_inductor/utils.py:1130] [9/7] DeviceCopy in input program 2024-05-19T19:55:02.9825179Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:55:02.9827646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:55:02.9829417Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:55:02.9829884Z 2024-05-19T19:55:05.9751050Z W0519 19:55:05.974000 140350158811776 torch/_inductor/utils.py:1130] [9/8] DeviceCopy in input program 2024-05-19T19:55:10.9053708Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:55:10.9057274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:55:10.9059320Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:55:10.9059858Z 2024-05-19T19:55:13.9035209Z W0519 19:55:13.902000 140350158811776 torch/_inductor/utils.py:1130] [9/9] DeviceCopy in input program 2024-05-19T19:55:18.3962943Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:55:18.3964860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:55:18.3966123Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:55:18.3966490Z 2024-05-19T19:55:21.8040900Z W0519 19:55:21.803000 140350158811776 torch/_inductor/utils.py:1130] [9/10] DeviceCopy in input program 2024-05-19T19:55:26.3217552Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:55:26.3219649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:55:26.3220911Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:55:26.3221227Z 2024-05-19T19:55:29.3298713Z W0519 19:55:29.329000 140350158811776 torch/_inductor/utils.py:1130] [9/11] DeviceCopy in input program 2024-05-19T19:55:34.2913763Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-05-19T19:55:34.2915505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_569 2024-05-19T19:55:34.2916700Z rand_attn = np.stack(rand_attn, axis=0) 2024-05-19T19:55:34.2917029Z 2024-05-19T19:56:00.5068274Z fail_accuracy 2024-05-19T19:56:00.5214466Z TIMING: entire_frame_compile:101.36152 code_gen:51.79953 inductor_compile:72.16213 backend_compile:90.83097 2024-05-19T19:56:00.5215789Z STATS: call_* op count: 3313 | FakeTensorMode.__torch_dispatch__:114828 | FakeTensor.__torch_dispatch__:9703 | ProxyTorchDispatchMode.__torch_dispatch__:30887 2024-05-19T19:56:00.5218948Z Dynamo produced 64 graphs covering 3313 ops with 46 graph breaks (4 unique) 2024-05-19T19:56:04.8806535Z 2024-05-19T19:56:06.6194990Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:56:06.6195685Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:56:06.6222788Z cuda eval hf_DistilBert 2024-05-19T19:56:18.5603017Z pass 2024-05-19T19:56:18.5816684Z TIMING: code_gen:2.43114 inductor_compile:4.20083 backend_compile:7.47281 entire_frame_compile:8.97867 2024-05-19T19:56:18.5817986Z STATS: call_* op count: 406 | FakeTensor.__torch_dispatch__:1910 | FakeTensorMode.__torch_dispatch__:18204 | ProxyTorchDispatchMode.__torch_dispatch__:6343 2024-05-19T19:56:18.5819115Z Dynamo produced 1 graphs covering 406 ops with 0 graph breaks (0 unique) 2024-05-19T19:56:22.0542314Z 2024-05-19T19:56:24.8714233Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:56:24.8714712Z loading model: 0it [00:02, ?it/s] 2024-05-19T19:56:24.8753568Z cuda eval hf_GPT2 2024-05-19T19:56:45.5399377Z pass 2024-05-19T19:56:45.5709020Z TIMING: code_gen:3.20029 inductor_compile:6.16097 backend_compile:11.85462 entire_frame_compile:15.93921 2024-05-19T19:56:45.5710843Z STATS: call_* op count: 1244 | FakeTensorMode.__torch_dispatch__:31079 | FakeTensor.__torch_dispatch__:2422 | ProxyTorchDispatchMode.__torch_dispatch__:10472 2024-05-19T19:56:45.5712271Z Dynamo produced 1 graphs covering 1244 ops with 0 graph breaks (0 unique) 2024-05-19T19:56:49.2768715Z 2024-05-19T19:57:01.2177498Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:57:01.2178177Z loading model: 0it [00:11, ?it/s] 2024-05-19T19:57:01.4024975Z cuda eval hf_GPT2_large 2024-05-19T19:57:01.4026961Z pass_due_to_skip 2024-05-19T19:57:01.7680783Z TIMING: 2024-05-19T19:57:01.7681109Z STATS: call_* op count: 0 2024-05-19T19:57:01.7681680Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-05-19T19:57:04.7247664Z 2024-05-19T19:57:05.6639471Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:57:05.6639948Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:57:05.6661312Z cuda eval hf_Reformer 2024-05-19T19:57:09.9711581Z W0519 19:57:09.970000 139821293494912 torch/_dynamo/variables/tensor.py:700] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-05-19T19:57:09.9712957Z W0519 19:57:09.970000 139821293494912 torch/_dynamo/variables/tensor.py:700] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-05-19T19:57:09.9714455Z W0519 19:57:09.970000 139821293494912 torch/_dynamo/variables/tensor.py:700] [5/0] or: 2024-05-19T19:57:09.9715651Z W0519 19:57:09.970000 139821293494912 torch/_dynamo/variables/tensor.py:700] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-05-19T19:57:09.9717170Z W0519 19:57:09.970000 139821293494912 torch/_dynamo/variables/tensor.py:700] [5/0] to include these operations in the captured graph. 2024-05-19T19:57:09.9718482Z W0519 19:57:09.970000 139821293494912 torch/_dynamo/variables/tensor.py:700] [5/0] 2024-05-19T19:57:23.4865944Z skipping cudagraphs due to ['incompatible ops'] 2024-05-19T19:57:25.8744047Z pass 2024-05-19T19:57:25.8786882Z TIMING: entire_frame_compile:15.78046 code_gen:8.44263 inductor_compile:11.0381 backend_compile:13.6338 2024-05-19T19:57:25.8788195Z STATS: call_* op count: 482 | FakeTensorMode.__torch_dispatch__:17317 | ProxyTorchDispatchMode.__torch_dispatch__:4889 | FakeTensor.__torch_dispatch__:1716 2024-05-19T19:57:25.8789336Z Dynamo produced 7 graphs covering 482 ops with 5 graph breaks (2 unique) 2024-05-19T19:57:29.5262825Z 2024-05-19T19:57:31.2312318Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:57:31.2313155Z loading model: 0it [00:01, ?it/s] 2024-05-19T19:57:31.2347525Z cuda eval hf_T5 2024-05-19T19:58:06.3629573Z pass 2024-05-19T19:58:06.4011513Z TIMING: code_gen:5.07486 inductor_compile:8.45159 backend_compile:15.81386 entire_frame_compile:23.23191 2024-05-19T19:58:06.4013396Z STATS: call_* op count: 1654 | FakeTensorMode.__torch_dispatch__:45747 | FakeTensor.__torch_dispatch__:2994 | ProxyTorchDispatchMode.__torch_dispatch__:15891 2024-05-19T19:58:06.4014807Z Dynamo produced 1 graphs covering 1654 ops with 0 graph breaks (0 unique) 2024-05-19T19:58:10.2140907Z 2024-05-19T19:58:13.9169138Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:58:13.9169754Z loading model: 0it [00:03, ?it/s] 2024-05-19T19:58:13.9170284Z cuda eval hf_T5_base 2024-05-19T19:58:13.9180692Z Traceback (most recent call last): 2024-05-19T19:58:13.9181566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2280, in validate_model 2024-05-19T19:58:13.9182461Z self.model_iter_fn(model, example_inputs) 2024-05-19T19:58:13.9184291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 426, in forward_pass 2024-05-19T19:58:13.9185049Z return mod(*inputs) 2024-05-19T19:58:13.9186658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:58:13.9187773Z return self._call_impl(*args, **kwargs) 2024-05-19T19:58:13.9189177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:58:13.9190230Z return forward_call(*args, **kwargs) 2024-05-19T19:58:13.9191828Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-05-19T19:58:13.9193240Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-05-19T19:58:13.9194533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:58:13.9195578Z return self._call_impl(*args, **kwargs) 2024-05-19T19:58:13.9196544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:58:13.9197382Z return forward_call(*args, **kwargs) 2024-05-19T19:58:13.9198677Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-05-19T19:58:13.9199832Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-05-19T19:58:13.9201011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:58:13.9201921Z return self._call_impl(*args, **kwargs) 2024-05-19T19:58:13.9202848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:58:13.9203693Z return forward_call(*args, **kwargs) 2024-05-19T19:58:13.9204665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1743, in forward 2024-05-19T19:58:13.9205540Z decoder_outputs = self.decoder( 2024-05-19T19:58:13.9206791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:58:13.9207699Z return self._call_impl(*args, **kwargs) 2024-05-19T19:58:13.9208636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:58:13.9209489Z return forward_call(*args, **kwargs) 2024-05-19T19:58:13.9210460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1110, in forward 2024-05-19T19:58:13.9211322Z layer_outputs = layer_module( 2024-05-19T19:58:13.9212260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:58:13.9213154Z return self._call_impl(*args, **kwargs) 2024-05-19T19:58:13.9214139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:58:13.9214985Z return forward_call(*args, **kwargs) 2024-05-19T19:58:13.9216043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 694, in forward 2024-05-19T19:58:13.9216937Z self_attention_outputs = self.layer[0]( 2024-05-19T19:58:13.9217931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:58:13.9218841Z return self._call_impl(*args, **kwargs) 2024-05-19T19:58:13.9219787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:58:13.9220638Z return forward_call(*args, **kwargs) 2024-05-19T19:58:13.9221611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 601, in forward 2024-05-19T19:58:13.9222506Z attention_output = self.SelfAttention( 2024-05-19T19:58:13.9223519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl 2024-05-19T19:58:13.9224448Z return self._call_impl(*args, **kwargs) 2024-05-19T19:58:13.9225385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl 2024-05-19T19:58:13.9226231Z return forward_call(*args, **kwargs) 2024-05-19T19:58:13.9227185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 561, in forward 2024-05-19T19:58:13.9228279Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2024-05-19T19:58:13.9231842Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 58.44 MiB is free. Process 11013 has 21.91 GiB memory in use. Of the allocated memory 20.94 GiB is allocated by PyTorch, and 666.49 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-05-19T19:58:13.9234557Z 2024-05-19T19:58:13.9234853Z The above exception was the direct cause of the following exception: 2024-05-19T19:58:13.9235304Z 2024-05-19T19:58:13.9235451Z Traceback (most recent call last): 2024-05-19T19:58:13.9236107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4109, in run 2024-05-19T19:58:13.9236769Z ) = runner.load_model( 2024-05-19T19:58:13.9237430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-05-19T19:58:13.9238413Z self.validate_model(model, example_inputs) 2024-05-19T19:58:13.9239180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2282, in validate_model 2024-05-19T19:58:13.9239961Z raise RuntimeError("Eager run failed") from e 2024-05-19T19:58:13.9240432Z RuntimeError: Eager run failed 2024-05-19T19:58:13.9240795Z 2024-05-19T19:58:13.9240921Z eager_fail_to_run 2024-05-19T19:58:16.8931327Z 2024-05-19T19:58:23.5606455Z loading model: 0it [00:00, ?it/s] 2024-05-19T19:58:23.5606962Z loading model: 0it [00:06, ?it/s] 2024-05-19T19:58:23.5641919Z cuda eval hf_T5_generate 2024-05-19T19:58:53.7541662Z W0519 19:58:53.753000 140639847121536 torch/_dynamo/variables/tensor.py:700] [4/0] Graph break from `Tensor.item()`, consider setting: 2024-05-19T19:58:53.7543343Z W0519 19:58:53.753000 140639847121536 torch/_dynamo/variables/tensor.py:700] [4/0] torch._dynamo.config.capture_scalar_outputs = True 2024-05-19T19:58:53.7544919Z W0519 19:58:53.753000 140639847121536 torch/_dynamo/variables/tensor.py:700] [4/0] or: 2024-05-19T19:58:53.7546346Z W0519 19:58:53.753000 140639847121536 torch/_dynamo/variables/tensor.py:700] [4/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-05-19T19:58:53.7547647Z W0519 19:58:53.753000 140639847121536 torch/_dynamo/variables/tensor.py:700] [4/0] to include these operations in the captured graph. 2024-05-19T19:58:53.7548707Z W0519 19:58:53.753000 140639847121536 torch/_dynamo/variables/tensor.py:700] [4/0] 2024-05-19T20:01:05.1809254Z W0519 20:01:05.180000 140639847121536 torch/_dynamo/convert_frame.py:752] [9/8] torch._dynamo hit config.cache_size_limit (8) 2024-05-19T20:01:05.1811700Z W0519 20:01:05.180000 140639847121536 torch/_dynamo/convert_frame.py:752] [9/8] function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1642) 2024-05-19T20:01:05.1813527Z W0519 20:01:05.180000 140639847121536 torch/_dynamo/convert_frame.py:752] [9/8] last reason: ___check_obj_id(L['past_key_values'], 7632672) 2024-05-19T20:01:05.1814962Z W0519 20:01:05.180000 140639847121536 torch/_dynamo/convert_frame.py:752] [9/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-05-19T20:01:05.1816627Z W0519 20:01:05.180000 140639847121536 torch/_dynamo/convert_frame.py:752] [9/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-05-19T20:03:14.1161757Z W0519 20:03:14.114000 140639847121536 torch/_dynamo/convert_frame.py:752] [15/8] torch._dynamo hit config.cache_size_limit (8) 2024-05-19T20:03:14.1163942Z W0519 20:03:14.114000 140639847121536 torch/_dynamo/convert_frame.py:752] [15/8] function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:973) 2024-05-19T20:03:14.1165821Z W0519 20:03:14.114000 140639847121536 torch/_dynamo/convert_frame.py:752] [15/8] last reason: tensor 'L['input_ids']' stride mismatch at index 0. expected 9, actual 17 2024-05-19T20:03:14.1167410Z W0519 20:03:14.114000 140639847121536 torch/_dynamo/convert_frame.py:752] [15/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-05-19T20:03:14.1169174Z W0519 20:03:14.114000 140639847121536 torch/_dynamo/convert_frame.py:752] [15/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-05-19T20:05:22.7057579Z pass 2024-05-19T20:05:22.7069546Z TIMING: entire_frame_compile:333.57831 code_gen:44.91541 inductor_compile:120.4422 backend_compile:241.47076 2024-05-19T20:05:22.7071595Z STATS: call_* op count: 11232 | FakeTensorMode.__torch_dispatch__:408454 | FakeTensor.__torch_dispatch__:36701 | ProxyTorchDispatchMode.__torch_dispatch__:107225 | attempt fast:2078 | fast is_contiguous:1454 | slow both tensors nontrivially broadcast:258 | slow no contiguity match:366 2024-05-19T20:05:22.7073410Z Dynamo produced 58 graphs covering 11232 ops with 5 graph breaks (3 unique) 2024-05-19T20:05:52.3958353Z 2024-05-19T20:06:01.7249820Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:06:01.7250256Z loading model: 0it [00:09, ?it/s] 2024-05-19T20:06:01.7685819Z cuda eval hf_T5_large 2024-05-19T20:06:01.7688838Z pass_due_to_skip 2024-05-19T20:06:02.0638746Z TIMING: 2024-05-19T20:06:02.0639075Z STATS: call_* op count: 0 2024-05-19T20:06:02.0641765Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-05-19T20:06:05.0512199Z 2024-05-19T20:06:05.9056883Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:06:05.9057496Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:06:05.9075194Z cuda eval hf_Whisper 2024-05-19T20:06:16.3075012Z pass 2024-05-19T20:06:16.3120514Z TIMING: code_gen:3.30602 inductor_compile:4.32418 backend_compile:6.68868 entire_frame_compile:7.54196 2024-05-19T20:06:16.3121833Z STATS: call_* op count: 230 | FakeTensor.__torch_dispatch__:1318 | FakeTensorMode.__torch_dispatch__:10562 | ProxyTorchDispatchMode.__torch_dispatch__:3561 2024-05-19T20:06:16.3122946Z Dynamo produced 1 graphs covering 230 ops with 0 graph breaks (0 unique) 2024-05-19T20:06:19.7493949Z 2024-05-19T20:06:24.2087033Z loading model: 0it [00:00, ?it/s] 2024-05-19T20:06:24.2087775Z loading model: 0it [00:04, ?it/s] 2024-05-19T20:06:24.2173827Z cuda eval hf_distil_whisper 2024-05-19T20:07:13.7048697Z pass 2024-05-19T20:07:13.7110133Z TIMING: code_gen:6.4059 inductor_compile:11.47904 backend_compile:20.427 entire_frame_compile:24.62461 2024-05-19T20:07:13.7111838Z STATS: call_* op count: 1270 | FakeTensor.__torch_dispatch__:7018 | FakeTensorMode.__torch_dispatch__:55760 | ProxyTorchDispatchMode.__torch_dispatch__:19621 2024-05-19T20:07:13.7112981Z Dynamo produced 1 graphs covering 1270 ops with 0 graph breaks (0 unique) 2024-05-19T20:07:15.4768592Z accuracy pass_rate=62.50% 2024-05-19T20:07:15.4771545Z calls_captured gmean=0.00x mean=729.396x 2024-05-19T20:07:15.4774348Z unique_graphs gmean=0.00x mean=4.875x 2024-05-19T20:07:15.4777549Z graph_breaks gmean=0.00x mean=2.062x 2024-05-19T20:07:15.4780774Z unique_graph_breaks gmean=0.00x mean=0.458x 2024-05-19T20:07:15.4784039Z autograd_captures gmean=0.00x mean=0.000x 2024-05-19T20:07:15.4787070Z autograd_compiles gmean=0.00x mean=0.000x 2024-05-19T20:07:15.4790290Z cudagraph_skips gmean=0.00x mean=0.521x 2024-05-19T20:07:16.2933988Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv 2024-05-19T20:07:16.6091424Z torchrec_dlrm XFAIL 2024-05-19T20:07:16.6095534Z BERT_pytorch PASS 2024-05-19T20:07:16.6101195Z Background_Matting XFAIL 2024-05-19T20:07:16.6106513Z DALLE2_pytorch FAIL: accuracy=eager_fail_to_run, expected=pass 2024-05-19T20:07:16.6111724Z LearningToPaint PASS 2024-05-19T20:07:16.6117284Z Super_SloMo PASS 2024-05-19T20:07:16.6123048Z alexnet PASS 2024-05-19T20:07:16.6127750Z basic_gnn_edgecnn PASS 2024-05-19T20:07:16.6133102Z basic_gnn_gcn PASS 2024-05-19T20:07:16.6138542Z basic_gnn_gin PASS 2024-05-19T20:07:16.6143680Z basic_gnn_sage PASS 2024-05-19T20:07:16.6148723Z cm3leon_generate PASS 2024-05-19T20:07:16.6153967Z dcgan PASS 2024-05-19T20:07:16.6159370Z demucs PASS 2024-05-19T20:07:16.6164405Z densenet121 PASS 2024-05-19T20:07:16.6169669Z detectron2_fasterrcnn_r_101_c4 XFAIL 2024-05-19T20:07:16.6174928Z detectron2_fasterrcnn_r_101_dc5 XFAIL 2024-05-19T20:07:16.6180172Z detectron2_fasterrcnn_r_101_fpn XFAIL 2024-05-19T20:07:16.6185224Z detectron2_fasterrcnn_r_50_c4 XFAIL 2024-05-19T20:07:16.6190487Z detectron2_fasterrcnn_r_50_dc5 XFAIL 2024-05-19T20:07:16.6195886Z detectron2_fasterrcnn_r_50_fpn XFAIL 2024-05-19T20:07:16.6201620Z detectron2_fcos_r_50_fpn PASS 2024-05-19T20:07:16.6206883Z detectron2_maskrcnn_r_101_c4 XFAIL 2024-05-19T20:07:16.6212141Z detectron2_maskrcnn_r_101_fpn XFAIL 2024-05-19T20:07:16.6217320Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-05-19T20:07:16.6222877Z detectron2_maskrcnn_r_50_fpn XFAIL 2024-05-19T20:07:16.6227707Z dlrm PASS 2024-05-19T20:07:16.6233126Z doctr_det_predictor PASS 2024-05-19T20:07:16.6238438Z doctr_reco_predictor PASS 2024-05-19T20:07:16.6243707Z drq PASS 2024-05-19T20:07:16.6249048Z fastNLP_Bert FAIL: accuracy=eager_fail_to_run, expected=pass 2024-05-19T20:07:16.6254048Z functorch_dp_cifar10 PASS 2024-05-19T20:07:16.6259144Z functorch_maml_omniglot PASS 2024-05-19T20:07:16.6264305Z hf_Albert PASS 2024-05-19T20:07:16.6269325Z hf_Bart PASS 2024-05-19T20:07:16.6274733Z hf_Bert PASS 2024-05-19T20:07:16.6280211Z hf_Bert_large PASS 2024-05-19T20:07:16.6285444Z hf_BigBird XFAIL 2024-05-19T20:07:16.6290608Z hf_DistilBert PASS 2024-05-19T20:07:16.6295742Z hf_GPT2 PASS 2024-05-19T20:07:16.6301091Z hf_GPT2_large XFAIL 2024-05-19T20:07:16.6306345Z hf_Reformer PASS 2024-05-19T20:07:16.6311532Z hf_T5 PASS 2024-05-19T20:07:16.6316843Z hf_T5_base XFAIL 2024-05-19T20:07:16.6322044Z hf_T5_generate PASS 2024-05-19T20:07:16.6327313Z hf_T5_large XFAIL 2024-05-19T20:07:16.6332369Z hf_Whisper PASS 2024-05-19T20:07:16.6337785Z hf_distil_whisper PASS 2024-05-19T20:07:16.6338186Z 2024-05-19T20:07:16.6342614Z Error: 2 models have accuracy status regressed: 2024-05-19T20:07:16.6343319Z DALLE2_pytorch fastNLP_Bert 2024-05-19T20:07:16.6343682Z 2024-05-19T20:07:16.6343690Z 2024-05-19T20:07:16.6344594Z If this change is expected, you can update `benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv` to reflect the new baseline. 2024-05-19T20:07:16.6346879Z from pytorch/pytorch root, run 2024-05-19T20:07:16.6347866Z `python benchmarks/dynamo/ci_expected_accuracy/update_expected.py c8c7ddfde57442cc28a394c385cab656ce9b1f5e` 2024-05-19T20:07:16.6348885Z and then `git add` the resulting local changes to expected CSVs to your commit. 2024-05-19T20:07:16.6349384Z 2024-05-19T20:07:16.6813043Z + cleanup_workspace 2024-05-19T20:07:16.6814086Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-05-19T20:07:16.6815347Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-05-19T20:07:16.6816338Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-05-19T20:07:16.6817037Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-05-19T20:07:16.6817907Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-05-19T20:07:16.6818867Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-05-19T20:07:16.6819634Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-05-19T20:07:17.2022469Z ##[error]Process completed with exit code 1. 2024-05-19T20:07:17.2105658Z Prepare all required actions 2024-05-19T20:07:17.2106102Z Getting action download info 2024-05-19T20:07:17.3580523Z ##[group]Run ./.github/actions/pytest-cache-upload 2024-05-19T20:07:17.3580979Z with: 2024-05-19T20:07:17.3581271Z cache_dir: .pytest_cache 2024-05-19T20:07:17.3581627Z shard: 1 2024-05-19T20:07:17.3581967Z sha: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T20:07:17.3582439Z test_config: inductor_torchbench 2024-05-19T20:07:17.3582975Z job_identifier: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T20:07:17.3583494Z env: 2024-05-19T20:07:17.3583769Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:17.3584226Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:17.3584968Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:17.3585605Z ##[endgroup] 2024-05-19T20:07:17.3624638Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-05-19T20:07:17.3625239Z with: 2024-05-19T20:07:17.3625508Z shell: bash 2024-05-19T20:07:17.3625814Z timeout_minutes: 5 2024-05-19T20:07:17.3626141Z max_attempts: 5 2024-05-19T20:07:17.3626460Z retry_wait_seconds: 30 2024-05-19T20:07:17.3626905Z command: set -eu python3 -m pip install boto3==1.19.12 2024-05-19T20:07:17.3627415Z polling_interval_seconds: 1 2024-05-19T20:07:17.3627794Z warning_on_retry: true 2024-05-19T20:07:17.3628143Z continue_on_error: false 2024-05-19T20:07:17.3628472Z env: 2024-05-19T20:07:17.3628749Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:17.3629197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:17.3629925Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:17.3630556Z ##[endgroup] 2024-05-19T20:07:17.6300215Z Defaulting to user installation because normal site-packages is not writeable 2024-05-19T20:07:18.8647531Z Collecting boto3==1.19.12 2024-05-19T20:07:18.8845521Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2024-05-19T20:07:20.3031094Z Collecting botocore<1.23.0,>=1.22.12 2024-05-19T20:07:20.3179972Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2024-05-19T20:07:20.4833449Z Collecting s3transfer<0.6.0,>=0.5.0 2024-05-19T20:07:20.4874862Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-05-19T20:07:20.5315632Z Collecting jmespath<1.0.0,>=0.7.1 2024-05-19T20:07:20.5358896Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-05-19T20:07:20.5438874Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.18) 2024-05-19T20:07:20.6052141Z Collecting python-dateutil<3.0.0,>=2.1 2024-05-19T20:07:20.6096097Z Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 2024-05-19T20:07:20.6554325Z Collecting six>=1.5 2024-05-19T20:07:20.6603852Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-05-19T20:07:20.7302745Z Installing collected packages: six, python-dateutil, jmespath, botocore, s3transfer, boto3 2024-05-19T20:07:21.4078192Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.9.0.post0 s3transfer-0.5.2 six-1.16.0 2024-05-19T20:07:22.4159626Z Command completed after 1 attempt(s). 2024-05-19T20:07:22.4204080Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2024-05-19T20:07:22.4204676Z python3 .github/scripts/pytest_cache.py \ 2024-05-19T20:07:22.4205158Z  --upload \ 2024-05-19T20:07:22.4205575Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2024-05-19T20:07:22.4206091Z  --pr_identifier $GITHUB_REF \ 2024-05-19T20:07:22.4206566Z  --job_identifier $JOB_IDENTIFIER \ 2024-05-19T20:07:22.4207012Z  --sha $SHA \ 2024-05-19T20:07:22.4207390Z  --test_config $TEST_CONFIG \ 2024-05-19T20:07:22.4207822Z  --shard $SHARD \ 2024-05-19T20:07:22.4208197Z  --repo $REPO \ 2024-05-19T20:07:22.4208703Z  --temp_dir $RUNNER_TEMP \ 2024-05-19T20:07:22.4216437Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:22.4216945Z env: 2024-05-19T20:07:22.4217228Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.4217689Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.4218435Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.4219153Z CACHE_DIR: .pytest_cache 2024-05-19T20:07:22.4219642Z JOB_IDENTIFIER: inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-05-19T20:07:22.4220240Z SHA: c8c7ddfde57442cc28a394c385cab656ce9b1f5e 2024-05-19T20:07:22.4220715Z TEST_CONFIG: inductor_torchbench 2024-05-19T20:07:22.4221103Z SHARD: 1 2024-05-19T20:07:22.4221397Z REPO: pytorch/pytorch 2024-05-19T20:07:22.4221737Z ##[endgroup] 2024-05-19T20:07:22.5527502Z PR identifier for `refs/tags/ciflow/inductor/121956` is `5ec4bd29b9652d20d61d9e0a665b8d46` 2024-05-19T20:07:22.5530702Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='inductor_linux-focal-cuda12.4-py3.10-gcc9-sm86', pr_identifier='refs/tags/ciflow/inductor/121956', repo='pytorch/pytorch', sha='c8c7ddfde57442cc28a394c385cab656ce9b1f5e', shard='1', temp_dir='/home/ec2-user/actions-runner/_work/_temp', test_config='inductor_torchbench', upload=True) 2024-05-19T20:07:22.5533593Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload 2024-05-19T20:07:22.5738765Z ##[group]Run cat test/**/*_toprint.log || true 2024-05-19T20:07:22.5739280Z cat test/**/*_toprint.log || true 2024-05-19T20:07:22.5746977Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:22.5747490Z env: 2024-05-19T20:07:22.5747777Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.5748241Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.5749143Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.5749811Z ##[endgroup] 2024-05-19T20:07:22.5812835Z cat: test/**/*_toprint.log: No such file or directory 2024-05-19T20:07:22.5861297Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-05-19T20:07:22.5861776Z kill "$MONITOR_SCRIPT_PID" 2024-05-19T20:07:22.5869300Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:22.5869796Z env: 2024-05-19T20:07:22.5870084Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.5870545Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.5871278Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.5872096Z MONITOR_SCRIPT_PID: 28814 2024-05-19T20:07:22.5872466Z ##[endgroup] 2024-05-19T20:07:22.6058167Z Prepare all required actions 2024-05-19T20:07:22.6058613Z Getting action download info 2024-05-19T20:07:22.7114035Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-05-19T20:07:22.8572353Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-05-19T20:07:22.8572820Z with: 2024-05-19T20:07:22.8573363Z file-suffix: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363 2024-05-19T20:07:22.8574034Z s3-bucket: gha-artifacts 2024-05-19T20:07:22.8574391Z env: 2024-05-19T20:07:22.8574674Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.8575141Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.8575893Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.8576537Z ##[endgroup] 2024-05-19T20:07:22.8602299Z ##[group]Run # Remove any previous test jsons if they exist 2024-05-19T20:07:22.8602958Z # Remove any previous test jsons if they exist 2024-05-19T20:07:22.8603478Z rm -f test-jsons-*.zip 2024-05-19T20:07:22.8604002Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-05-19T20:07:22.8611814Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:22.8612319Z env: 2024-05-19T20:07:22.8612607Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.8613059Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.8613804Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.8614699Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363 2024-05-19T20:07:22.8615344Z ##[endgroup] 2024-05-19T20:07:22.8812398Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-05-19T20:07:22.8842004Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-05-19T20:07:22.8842710Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-05-19T20:07:22.8848820Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-05-19T20:07:22.8875123Z ##[group]Run # Remove any previous test reports if they exist 2024-05-19T20:07:22.8875793Z # Remove any previous test reports if they exist 2024-05-19T20:07:22.8876329Z rm -f test-reports-*.zip 2024-05-19T20:07:22.8876931Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-05-19T20:07:22.8884664Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:22.8885174Z env: 2024-05-19T20:07:22.8885469Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.8885939Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.8886696Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.8887598Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363 2024-05-19T20:07:22.8888252Z ##[endgroup] 2024-05-19T20:07:22.9049364Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-05-19T20:07:22.9055122Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 95%) 2024-05-19T20:07:22.9057044Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 80%) 2024-05-19T20:07:22.9083853Z ##[group]Run # Remove any previous usage logs if they exist 2024-05-19T20:07:22.9084500Z # Remove any previous usage logs if they exist 2024-05-19T20:07:22.9085007Z rm -f logs-*.zip 2024-05-19T20:07:22.9085677Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-05-19T20:07:22.9086467Z # so check to see if the file exists first 2024-05-19T20:07:22.9086974Z if [ -f 'usage_log.txt' ]; then 2024-05-19T20:07:22.9087511Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-05-19T20:07:22.9088008Z fi 2024-05-19T20:07:22.9088380Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-05-19T20:07:22.9089064Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-05-19T20:07:22.9089571Z fi 2024-05-19T20:07:22.9096903Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:22.9097405Z env: 2024-05-19T20:07:22.9097692Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.9098163Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.9098962Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.9099858Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363 2024-05-19T20:07:22.9100499Z ##[endgroup] 2024-05-19T20:07:22.9277245Z adding: usage_log.txt (deflated 96%) 2024-05-19T20:07:22.9342493Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-05-19T20:07:22.9343222Z # Remove any previous debugging artifacts if they exist 2024-05-19T20:07:22.9343779Z rm -f debug-*.zip 2024-05-19T20:07:22.9344179Z if [ -d 'test/debug' ]; then 2024-05-19T20:07:22.9344688Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-05-19T20:07:22.9345177Z fi 2024-05-19T20:07:22.9351960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:22.9352484Z env: 2024-05-19T20:07:22.9352773Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.9353239Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.9353980Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.9354880Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363 2024-05-19T20:07:22.9355518Z ##[endgroup] 2024-05-19T20:07:22.9445023Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:07:22.9445460Z with: 2024-05-19T20:07:22.9445742Z s3-bucket: gha-artifacts 2024-05-19T20:07:22.9446172Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:07:22.9446644Z retention-days: 14 2024-05-19T20:07:22.9446986Z if-no-files-found: warn 2024-05-19T20:07:22.9447345Z path: test-jsons-*.zip 2024-05-19T20:07:22.9447689Z name: artifact 2024-05-19T20:07:22.9447997Z region: us-east-1 2024-05-19T20:07:22.9448303Z env: 2024-05-19T20:07:22.9448584Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:22.9449034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:22.9449772Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:22.9450406Z ##[endgroup] 2024-05-19T20:07:23.2550615Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T20:07:23.2551221Z With the provided path, there will be 1 file uploaded 2024-05-19T20:07:23.2551925Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:07:23.4068901Z Starting upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363.zip 2024-05-19T20:07:23.5223311Z Finished upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363.zip 2024-05-19T20:07:23.5365595Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:07:23.5366035Z with: 2024-05-19T20:07:23.5366330Z s3-bucket: gha-artifacts 2024-05-19T20:07:23.5366905Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:07:23.5367394Z retention-days: 14 2024-05-19T20:07:23.5367736Z if-no-files-found: error 2024-05-19T20:07:23.5368110Z path: test-reports-*.zip 2024-05-19T20:07:23.5368465Z name: artifact 2024-05-19T20:07:23.5368779Z region: us-east-1 2024-05-19T20:07:23.5369096Z env: 2024-05-19T20:07:23.5369384Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:23.5369846Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:23.5370582Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:23.5371227Z ##[endgroup] 2024-05-19T20:07:23.8474083Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T20:07:23.8474865Z With the provided path, there will be 1 file uploaded 2024-05-19T20:07:23.8475797Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:07:23.8500993Z Starting upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363.zip 2024-05-19T20:07:23.9412456Z Finished upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363.zip 2024-05-19T20:07:23.9558386Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:07:23.9558833Z with: 2024-05-19T20:07:23.9559130Z s3-bucket: gha-artifacts 2024-05-19T20:07:23.9559567Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:07:23.9560055Z retention-days: 14 2024-05-19T20:07:23.9560403Z if-no-files-found: ignore 2024-05-19T20:07:23.9560780Z path: logs-*.zip 2024-05-19T20:07:23.9561101Z name: artifact 2024-05-19T20:07:23.9561411Z region: us-east-1 2024-05-19T20:07:23.9561724Z env: 2024-05-19T20:07:23.9562020Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:23.9562487Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:23.9563246Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:23.9563898Z ##[endgroup] 2024-05-19T20:07:24.2660942Z NOTE: s3-prefix specified, ignoring name parameter 2024-05-19T20:07:24.2661660Z With the provided path, there will be 1 file uploaded 2024-05-19T20:07:24.2662289Z Uploading to s3 prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:07:24.2687482Z Starting upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363.zip 2024-05-19T20:07:24.3827958Z Finished upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_25153197363.zip 2024-05-19T20:07:24.3971227Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:07:24.3971669Z with: 2024-05-19T20:07:24.3971972Z s3-bucket: gha-artifacts 2024-05-19T20:07:24.3972413Z s3-prefix: pytorch/pytorch/9148947323/1/artifact 2024-05-19T20:07:24.3972897Z retention-days: 14 2024-05-19T20:07:24.3973244Z if-no-files-found: ignore 2024-05-19T20:07:24.3973626Z path: debug-*.zip 2024-05-19T20:07:24.3973951Z name: artifact 2024-05-19T20:07:24.3974258Z region: us-east-1 2024-05-19T20:07:24.3974563Z env: 2024-05-19T20:07:24.3974864Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:24.3975331Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:24.3976070Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:24.3976720Z ##[endgroup] 2024-05-19T20:07:24.7112324Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-05-19T20:07:24.7252697Z ##[group]Run # shellcheck disable=SC2156 2024-05-19T20:07:24.7253178Z # shellcheck disable=SC2156 2024-05-19T20:07:24.7253971Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-05-19T20:07:24.7261881Z shell: /usr/bin/bash -e {0} 2024-05-19T20:07:24.7262240Z env: 2024-05-19T20:07:24.7262522Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:24.7262983Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:24.7263710Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:24.7264464Z ##[endgroup] 2024-05-19T20:07:24.9075719Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-05-19T20:07:24.9076234Z with: 2024-05-19T20:07:24.9076701Z name: coredumps-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu 2024-05-19T20:07:24.9077285Z retention-days: 14 2024-05-19T20:07:24.9077838Z if-no-files-found: ignore 2024-05-19T20:07:24.9078329Z path: ./**/core.[1-9]* 2024-05-19T20:07:24.9078780Z s3-bucket: gha-artifacts 2024-05-19T20:07:24.9079228Z region: us-east-1 2024-05-19T20:07:24.9079581Z env: 2024-05-19T20:07:24.9079889Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:24.9080342Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:24.9081070Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:24.9081698Z ##[endgroup] 2024-05-19T20:07:34.8306652Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2024-05-19T20:07:34.8540642Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-05-19T20:07:34.8541203Z with: 2024-05-19T20:07:34.8541463Z env: 2024-05-19T20:07:34.8541742Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:34.8542197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:34.8542925Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:34.8543562Z ##[endgroup] 2024-05-19T20:07:34.8564076Z ##[group]Run set -eou pipefail 2024-05-19T20:07:34.8564488Z set -eou pipefail 2024-05-19T20:07:34.8565003Z  2024-05-19T20:07:34.8565517Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-05-19T20:07:34.8566160Z for _ in $(seq 1440); do 2024-05-19T20:07:34.8566640Z  # Break if no ssh session exists anymore 2024-05-19T20:07:34.8567144Z  if [ "$(who)" = "" ]; then 2024-05-19T20:07:34.8567554Z  break 2024-05-19T20:07:34.8567915Z  fi 2024-05-19T20:07:34.8568222Z  echo "." 2024-05-19T20:07:34.8568557Z  sleep 5 2024-05-19T20:07:34.8568882Z done 2024-05-19T20:07:34.8576574Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:34.8577069Z env: 2024-05-19T20:07:34.8577367Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:34.8577827Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:34.8578572Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:34.8579203Z ##[endgroup] 2024-05-19T20:07:34.8600911Z Holding runner for 2 hours until all ssh sessions have logged out 2024-05-19T20:07:34.8651406Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T20:07:34.8652165Z # ignore expansion of "docker ps -q" since it could be empty 2024-05-19T20:07:34.8652760Z # shellcheck disable=SC2046 2024-05-19T20:07:34.8653249Z docker stop $(docker ps -q) || true 2024-05-19T20:07:34.8653743Z # Prune all of the docker images 2024-05-19T20:07:34.8654204Z docker system prune -af 2024-05-19T20:07:34.8661641Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:34.8662144Z env: 2024-05-19T20:07:34.8662431Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:34.8662891Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:34.8663626Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:34.8664264Z ##[endgroup] 2024-05-19T20:07:35.4423373Z 71bd051df825 2024-05-19T20:07:37.5032187Z Deleted Containers: 2024-05-19T20:07:37.5032909Z 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:37.5033452Z 2024-05-19T20:07:41.4209656Z Deleted Images: 2024-05-19T20:07:41.4211284Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks:d55d6a6f08250cd2609d528116ab6fd6e787bee0 2024-05-19T20:07:41.4213801Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn8-py3-gcc9-inductor-benchmarks@sha256:66f062925f91f9b9204b1dc6d68caf8c89fa82fd4b12cb5f24c19aa4b1c87506 2024-05-19T20:07:41.4215349Z deleted: sha256:a0542ac5317cc182e1aac2cb8d63a2d6287450937722bab68464513bbbc603ef 2024-05-19T20:07:41.4216239Z deleted: sha256:1a656f83b702d1cdc805b9509e7194f6ca9a24882785be5289f8bc566c1c1a0b 2024-05-19T20:07:41.4217088Z deleted: sha256:5b7ca0efbb8f2f6ed9ac2e375c4a52f07e2bd16e65d1914a58dcfb56ab8b1623 2024-05-19T20:07:41.4217957Z deleted: sha256:6d2255f7483ad8cfa448dc385ff2880171aacd4db919fbf38e2b3c4580f80e1e 2024-05-19T20:07:41.4218797Z deleted: sha256:a39c366330a4b21f0585d02ed65f5beb66bd03e36471ebb255c5f7766f29c118 2024-05-19T20:07:41.4219638Z deleted: sha256:5b2e96a220babb6e3a9b843ce53d3f2015dd0e505fed00a447cbe551b22d329e 2024-05-19T20:07:41.4220479Z deleted: sha256:d7247668eca42c13d2c6440886a9b43b633e5cb9304e3e3f85ac8803de7dd873 2024-05-19T20:07:41.4221425Z deleted: sha256:989398e2b08b3b7c9bf03cdbdaa903261d145b7d45bc7c1fa3a1a4a54a7ced68 2024-05-19T20:07:41.4222359Z deleted: sha256:5c2558f875db144549ed0efa5deb36d738647f1c172a9628a34677fddb6a96ee 2024-05-19T20:07:41.4223288Z deleted: sha256:b56f2c08b00650c89ac0f4d07686ba8ea22d64f666d27e8be716aeae4e042f87 2024-05-19T20:07:41.4224205Z deleted: sha256:35689f96fd20d72b6c5fd24a65e3c52c46359609c897b0552d8b36e4d62f31d9 2024-05-19T20:07:41.4225119Z deleted: sha256:8a251e66682a1ad41addcf4890ffb62aa8b85f5d9f17d433e828f170339a767f 2024-05-19T20:07:41.4226049Z deleted: sha256:9482b91acd78f655bcd9a0a6ade0f1958caa1a8274b13f7366f830fa7b4ca16e 2024-05-19T20:07:41.4226985Z deleted: sha256:1f1bf62bcfe25406d5e8f3a48198af80782c6a996bcafaae53575bcc843bd50a 2024-05-19T20:07:41.4227918Z deleted: sha256:6b680df6a8beceb613eb411958551bf0368a6d4a932eead77896d41ff535946a 2024-05-19T20:07:41.4228867Z deleted: sha256:7aa1322c73a827485b091076ffa2dc207e0a900941ca2eb2704442ca7c23ed7d 2024-05-19T20:07:41.4229734Z deleted: sha256:f0d92b27a23be6ff6a30b0974841e39e2ca9cb761797d5cb3ee5676708fbfddc 2024-05-19T20:07:41.4230830Z deleted: sha256:15e17f6dcd9e56d48cc4d19f06cf6d42e155d9dcb5d6d3289f432b1e1adb78c2 2024-05-19T20:07:41.4231779Z deleted: sha256:a9363d4f861657188bb086df9f77430971cf00513d7f3c151d954bba04092dd4 2024-05-19T20:07:41.4232617Z deleted: sha256:afad6ce30f8cd2b2fc987fb31626f14465a034dd85cea22a30e012f7b50e831c 2024-05-19T20:07:41.4233485Z deleted: sha256:a17532a675e4ec56db1fc54e2f285986b4333c89eaed6520edb8c85209ef76ea 2024-05-19T20:07:41.4234359Z deleted: sha256:55ff1a7a2aa478a7a6c00675e825d39cea2c5955f3f318832fa06cf293ddb137 2024-05-19T20:07:41.4235198Z deleted: sha256:21b7839e8103541d09ad0af189e5ecbf9df69cc798831eada41d8ce869555e94 2024-05-19T20:07:41.4236030Z deleted: sha256:3f93640e4b85ecbb5cb4239715bd521fe05dac28e2e25e53b71b808bbc193590 2024-05-19T20:07:41.4236877Z deleted: sha256:887d227bae679856dd096dad895dbcf90b807f5c82c15ff17ec5a58bf6dac184 2024-05-19T20:07:41.4238023Z deleted: sha256:d2beb8f0ce120b86322d41f423b7f42e64fdb9e4ff5d5d912252849b8146b293 2024-05-19T20:07:41.4238884Z deleted: sha256:e41d38911c997f96258de8880c0251dee4bf259158eb081eafc17f32f8948763 2024-05-19T20:07:41.4239721Z deleted: sha256:b85b0e100a5a15d221dbbccb1e864e367d917d7a79fa199c15db2c1d9a52c92a 2024-05-19T20:07:41.4240543Z deleted: sha256:75258ce71d03c6b0525b6f842fa5efd1738dbe82b883613717fcb707a60621a3 2024-05-19T20:07:41.4241371Z deleted: sha256:a12c28acec35eb76da2791220f40353777e633b6cd7d4fd187c3954e03406c46 2024-05-19T20:07:41.4242197Z deleted: sha256:eaf53ac56a05ff328d594fda97812e9db215cf74b87965669e54d4c6c1669cb3 2024-05-19T20:07:41.4243023Z deleted: sha256:795e72492c71c7cce01ac9e0408bc22fa8f7da1e41614874b812afc49c93fabf 2024-05-19T20:07:41.4243838Z deleted: sha256:1d1e16daa70a0b1c04b023ebbbe56366e3b0cbb9147851a544b43581d66ab50c 2024-05-19T20:07:41.4244676Z deleted: sha256:7752b0e90516dde8d6b7499bd8261515a2eb4063ca5d381313ea697291402d42 2024-05-19T20:07:41.4245491Z deleted: sha256:13082c1d2da03c75509cd19b1e7d211d98b359b41d53223e861fea8ec6be2366 2024-05-19T20:07:41.4246448Z deleted: sha256:fef51b8c2cf2579b04f0d82fba989e43cdd93a484741e14588a942c728328c80 2024-05-19T20:07:41.4247263Z deleted: sha256:cfd02989b5663a4fc04e8468da08dbb6202dd256e851a6e30e7370818fa65830 2024-05-19T20:07:41.4248092Z deleted: sha256:7b3853f547cd41caf732c9bf113d7f11db7fb026f979a50cf7a56ab673d8a2d9 2024-05-19T20:07:41.4248923Z deleted: sha256:8eb59cbbd7ea753bff3586c088340d20ca32887792f3f68c5b2da7d57d11447c 2024-05-19T20:07:41.4249763Z deleted: sha256:d4eb9793eb3f33e1fc18c62f8a4765625d5e2accaffffeb52543799bfe45315a 2024-05-19T20:07:41.4250586Z deleted: sha256:9b388bd2595f78854fce5ceb1a6c673a7bf3f50f84f6350351171599fcad68c4 2024-05-19T20:07:41.4251402Z deleted: sha256:1443f9602fff883313532012c5506b56993ec95031e611cace0b38b3e215413b 2024-05-19T20:07:41.4252192Z deleted: sha256:7d5157e7800e8039c647490628f0b8f64c7ed0017070e69ed1200ae786c318d2 2024-05-19T20:07:41.4252982Z deleted: sha256:3b8e4334d22e46371d21e1d80991345f300244c4932c1687b2723408f8fe213c 2024-05-19T20:07:41.4253871Z deleted: sha256:bc9b902883a611c9653e47dfcdce6f88287f0ce083995cd17612e1ee52d02f45 2024-05-19T20:07:41.4254702Z deleted: sha256:87241399dcaeb6b49174c5033c7aff0f1e914591bd2a727a80fcd0e453b1bdbc 2024-05-19T20:07:41.4255529Z deleted: sha256:9b60d0bfab86b98267697851c8871dae33fc146608b8af9764f5e3d3e459c75b 2024-05-19T20:07:41.4256360Z deleted: sha256:ad76dc292ef5f88364d52bdd2e83680e697b28b2f6df337cd9c829ab007217e1 2024-05-19T20:07:41.4257195Z deleted: sha256:7a4baeb5479a531e458c9fa438d16143efc760acf3c997b1fed3129f2d8d2fc7 2024-05-19T20:07:41.4258209Z deleted: sha256:e682379b3ae2d005fd368b974d3314ece2598d1a6928ae7341bb89f3c79e8e03 2024-05-19T20:07:41.4259060Z deleted: sha256:d8d802f39c7d9d7ddd8f5acef34ee7e07fac7f84fa106255d5228d659daf1073 2024-05-19T20:07:41.4259906Z deleted: sha256:9f893b39e0b2a09fdee21eec466f95f637ccd2e77ab254c00eeca60d4f1a4671 2024-05-19T20:07:41.4260760Z deleted: sha256:01d8e759bdd0bbea54aea263352ffd962e77eeab5d109a7fa8e279f2efdb6719 2024-05-19T20:07:41.4261611Z deleted: sha256:cd0863a43a1691ccbba624e9d63b85a341adb8bc16335dec1291e5a04147b040 2024-05-19T20:07:41.4262476Z deleted: sha256:5fec222edb2dc032d9a5203274fbd944dfb9494597fbb76b063b89ac7e84273f 2024-05-19T20:07:41.4263316Z deleted: sha256:742f8005153dd61c5f7718e2b2a8b3f934780866ef56f7ca11a48088dc638b97 2024-05-19T20:07:41.4264202Z deleted: sha256:e670dd8df73c884bf8257489df70a65828a55222a493f717eca2b9df1f3e41fa 2024-05-19T20:07:41.4265045Z deleted: sha256:e09fa8af11b0b73b35ad8f1f6ec0f0f0bf8a05ed64aa9d530377ddd2e1f94b56 2024-05-19T20:07:41.4265889Z deleted: sha256:02c8d457f61de51676900b87bd530f776b761f2b8dad7f345152b256d7bb7864 2024-05-19T20:07:41.4266721Z deleted: sha256:433544e4353c9e0269dc71fa16c00ddcecdcd5676ed0df618c7b51ac78e23053 2024-05-19T20:07:41.4267580Z deleted: sha256:fc24af9fd5121adfbc7b75df0abb1ef8c2d3868f33b7b6b1e8bf4e214fac4bd6 2024-05-19T20:07:41.4268431Z deleted: sha256:ce1492ad94b71204f8b3fb8903d0ad48a96652635e7fc9347e430639ca15a737 2024-05-19T20:07:41.4269258Z deleted: sha256:96f2ae9435e6e12f68a88dc404c8079b2796639589970c34894a86ccff3f3732 2024-05-19T20:07:41.4270106Z deleted: sha256:5310d2bc0952e7c6d8cbade7ecafd88ce3a04d9da303fa6419583e13a02b58a9 2024-05-19T20:07:41.4270945Z deleted: sha256:099076d608a55a027d60f5b572a3991a8d690a01500acf2c696933a7e6d38650 2024-05-19T20:07:41.4271892Z deleted: sha256:862b05d0f8f7e3dff2d04bd49784a9901f4e22c5dd6544ae72eb3be05e23fffe 2024-05-19T20:07:41.4272732Z deleted: sha256:f7a888e5f7904827f6d71cebcba5169dbc7a78b20f228b9b9b625f38e0c52f24 2024-05-19T20:07:41.4273581Z deleted: sha256:ab9f75a4a7a636aae480f630eaddc05133dda64f1c42445b617c97d3d209991c 2024-05-19T20:07:41.4274439Z deleted: sha256:56dda0c9cee64db5c10d5dd5f08c6f6707263f5ee98e8a075d635e5be996e855 2024-05-19T20:07:41.4275294Z deleted: sha256:5faf9c0a9efe4675ecd21a4ec417d51077d5e75da9e673161a94e7d6cd43f92c 2024-05-19T20:07:41.4275804Z 2024-05-19T20:07:41.4275956Z Total reclaimed space: 41.64GB 2024-05-19T20:07:41.4308960Z ##[group]Run set +e 2024-05-19T20:07:41.4309356Z set +e 2024-05-19T20:07:41.4309660Z set -x 2024-05-19T20:07:41.4310056Z  2024-05-19T20:07:41.4310344Z nvidia-smi 2024-05-19T20:07:41.4310967Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2024-05-19T20:07:41.4312080Z # the case where the driver has already crashed as it still can get the driver version 2024-05-19T20:07:41.4313037Z # and some basic information like the bus ID. However, the rest of the information 2024-05-19T20:07:41.4313821Z # would be missing (ERR!), for example: 2024-05-19T20:07:41.4314275Z # 2024-05-19T20:07:41.4314693Z # +-----------------------------------------------------------------------------+ 2024-05-19T20:07:41.4315461Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2024-05-19T20:07:41.4316248Z # |-------------------------------+----------------------+----------------------+ 2024-05-19T20:07:41.4317008Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T20:07:41.4318131Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2024-05-19T20:07:41.4318885Z # | | | MIG M. | 2024-05-19T20:07:41.4319477Z # |===============================+======================+======================| 2024-05-19T20:07:41.4320122Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2024-05-19T20:07:41.4320873Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2024-05-19T20:07:41.4321567Z # | | | ERR! | 2024-05-19T20:07:41.4322200Z # +-------------------------------+----------------------+----------------------+ 2024-05-19T20:07:41.4322718Z # 2024-05-19T20:07:41.4323143Z # +-----------------------------------------------------------------------------+ 2024-05-19T20:07:41.4323803Z # | Processes: | 2024-05-19T20:07:41.4324533Z # | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T20:07:41.4325251Z # | ID ID Usage | 2024-05-19T20:07:41.4325862Z # |=============================================================================| 2024-05-19T20:07:41.4326482Z # +-----------------------------------------------------------------------------+ 2024-05-19T20:07:41.4326993Z # 2024-05-19T20:07:41.4327552Z # This should be reported as a failure instead as it will guarantee to fail when 2024-05-19T20:07:41.4328280Z # Docker tries to run with --gpus all 2024-05-19T20:07:41.4328724Z # 2024-05-19T20:07:41.4329260Z # So, the correct check here is to query one of the missing piece of info like 2024-05-19T20:07:41.4330040Z # GPU name, so that the command can fail accordingly 2024-05-19T20:07:41.4330732Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T20:07:41.4331314Z NVIDIA_SMI_STATUS=$? 2024-05-19T20:07:41.4331681Z  2024-05-19T20:07:41.4332301Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T20:07:41.4333216Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T20:07:41.4334046Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T20:07:41.4334757Z  .github/scripts/stop_runner_service.sh 2024-05-19T20:07:41.4335215Z fi 2024-05-19T20:07:41.4335508Z  2024-05-19T20:07:41.4336206Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2024-05-19T20:07:41.4337176Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2024-05-19T20:07:41.4338040Z # https://github.com/pytorch/test-infra/issues/4000 2024-05-19T20:07:41.4338646Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2024-05-19T20:07:41.4339138Z NVIDIA_SMI_STATUS=$? 2024-05-19T20:07:41.4339510Z  2024-05-19T20:07:41.4340122Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2024-05-19T20:07:41.4341041Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2024-05-19T20:07:41.4341865Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2024-05-19T20:07:41.4342583Z  .github/scripts/stop_runner_service.sh 2024-05-19T20:07:41.4343041Z fi 2024-05-19T20:07:41.4343341Z  2024-05-19T20:07:41.4343703Z # Check the GPU count to be a power of 2 2024-05-19T20:07:41.4344550Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2024-05-19T20:07:41.4345646Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2024-05-19T20:07:41.4346447Z  .github/scripts/stop_runner_service.sh 2024-05-19T20:07:41.4346907Z fi 2024-05-19T20:07:41.4354566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-05-19T20:07:41.4355070Z env: 2024-05-19T20:07:41.4355355Z GIT_DEFAULT_BRANCH: main 2024-05-19T20:07:41.4355819Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-05-19T20:07:41.4356558Z DOCKER_CONTAINER_ID: 71bd051df82520b44379dc3d412f5d1a305ff935521f45b89e43ef41f850e634 2024-05-19T20:07:41.4357338Z RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch 2024-05-19T20:07:41.4358086Z ##[endgroup] 2024-05-19T20:07:41.4381681Z + nvidia-smi 2024-05-19T20:07:42.9931098Z Sun May 19 20:07:42 2024 2024-05-19T20:07:42.9932083Z +-----------------------------------------------------------------------------------------+ 2024-05-19T20:07:42.9933033Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-05-19T20:07:42.9933819Z |-----------------------------------------+------------------------+----------------------+ 2024-05-19T20:07:42.9934619Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-05-19T20:07:42.9935515Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-05-19T20:07:42.9936220Z | | | MIG M. | 2024-05-19T20:07:42.9936769Z |=========================================+========================+======================| 2024-05-19T20:07:43.0059801Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-05-19T20:07:43.0061793Z | 0% 29C P0 59W / 300W | 0MiB / 23028MiB | 0% Default | 2024-05-19T20:07:43.0063520Z | | | N/A | 2024-05-19T20:07:43.0064717Z +-----------------------------------------+------------------------+----------------------+ 2024-05-19T20:07:43.0065291Z 2024-05-19T20:07:43.0065948Z +-----------------------------------------------------------------------------------------+ 2024-05-19T20:07:43.0066580Z | Processes: | 2024-05-19T20:07:43.0067287Z | GPU GI CI PID Type Process name GPU Memory | 2024-05-19T20:07:43.0068259Z | ID ID Usage | 2024-05-19T20:07:43.0068830Z |=========================================================================================| 2024-05-19T20:07:43.0069601Z | No running processes found | 2024-05-19T20:07:43.0070354Z +-----------------------------------------------------------------------------------------+ 2024-05-19T20:07:43.6285033Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-05-19T20:07:45.1900157Z NVIDIA A10G 2024-05-19T20:07:45.6320043Z + NVIDIA_SMI_STATUS=0 2024-05-19T20:07:45.6320616Z + '[' 0 -ne 0 ']' 2024-05-19T20:07:45.6323673Z ++ wc -l 2024-05-19T20:07:45.6324026Z ++ nvidia-smi --list-gpus 2024-05-19T20:07:47.6374054Z + GPU_COUNT=1 2024-05-19T20:07:47.6374650Z + NVIDIA_SMI_STATUS=0 2024-05-19T20:07:47.6375247Z + '[' 0 -ne 0 ']' 2024-05-19T20:07:47.6375688Z + '[' 1 -le 8 ']' 2024-05-19T20:07:47.6376027Z + '[' 1 -ne 1 ']' 2024-05-19T20:07:47.6432074Z Post job cleanup. 2024-05-19T20:07:47.6481901Z Post job cleanup. 2024-05-19T20:07:47.7308274Z [command]/usr/bin/git version 2024-05-19T20:07:47.7349056Z git version 2.40.1 2024-05-19T20:07:47.7390505Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4d639b64-c221-42d0-a75c-0198cdc39a1a' before making global git config changes 2024-05-19T20:07:47.7391874Z Adding repository directory to the temporary git global config as a safe directory 2024-05-19T20:07:47.7396380Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-05-19T20:07:47.7423113Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-05-19T20:07:47.7452535Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-05-19T20:07:47.7671425Z Entering 'android/libs/fbjni' 2024-05-19T20:07:47.7706464Z Entering 'third_party/FP16' 2024-05-19T20:07:47.7742766Z Entering 'third_party/FXdiv' 2024-05-19T20:07:47.7779258Z Entering 'third_party/NNPACK' 2024-05-19T20:07:47.7815698Z Entering 'third_party/QNNPACK' 2024-05-19T20:07:47.7851541Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T20:07:47.7888354Z Entering 'third_party/XNNPACK' 2024-05-19T20:07:47.7935747Z Entering 'third_party/benchmark' 2024-05-19T20:07:47.7967799Z Entering 'third_party/cpp-httplib' 2024-05-19T20:07:47.8004635Z Entering 'third_party/cpuinfo' 2024-05-19T20:07:47.8040160Z Entering 'third_party/cudnn_frontend' 2024-05-19T20:07:47.8075765Z Entering 'third_party/cutlass' 2024-05-19T20:07:47.8118094Z Entering 'third_party/eigen' 2024-05-19T20:07:47.8156130Z Entering 'third_party/fbgemm' 2024-05-19T20:07:47.8191821Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T20:07:47.8226854Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T20:07:47.8261209Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T20:07:47.8300980Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T20:07:47.8334397Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T20:07:47.8370679Z Entering 'third_party/flatbuffers' 2024-05-19T20:07:47.8409501Z Entering 'third_party/fmt' 2024-05-19T20:07:47.8441763Z Entering 'third_party/foxi' 2024-05-19T20:07:47.8476935Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T20:07:47.8510716Z Entering 'third_party/gloo' 2024-05-19T20:07:47.8546076Z Entering 'third_party/googletest' 2024-05-19T20:07:47.8582209Z Entering 'third_party/ideep' 2024-05-19T20:07:47.8616816Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T20:07:47.8657893Z Entering 'third_party/ios-cmake' 2024-05-19T20:07:47.8691898Z Entering 'third_party/ittapi' 2024-05-19T20:07:47.8726430Z Entering 'third_party/kineto' 2024-05-19T20:07:47.8759121Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T20:07:47.8792882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T20:07:47.8827098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T20:07:47.8863470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T20:07:47.8897663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T20:07:47.8932879Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T20:07:47.8970795Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T20:07:47.9005104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T20:07:47.9038597Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T20:07:47.9072510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T20:07:47.9107744Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T20:07:47.9143109Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T20:07:47.9179431Z Entering 'third_party/mimalloc' 2024-05-19T20:07:47.9214395Z Entering 'third_party/nccl/nccl' 2024-05-19T20:07:47.9248784Z Entering 'third_party/neon2sse' 2024-05-19T20:07:47.9283933Z Entering 'third_party/nlohmann' 2024-05-19T20:07:47.9317204Z Entering 'third_party/onnx' 2024-05-19T20:07:47.9363828Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T20:07:47.9398526Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T20:07:47.9435848Z Entering 'third_party/onnx-tensorrt' 2024-05-19T20:07:47.9471312Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T20:07:47.9508496Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T20:07:47.9546693Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T20:07:47.9582208Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T20:07:47.9622577Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T20:07:47.9657660Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T20:07:47.9692895Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T20:07:47.9726385Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T20:07:47.9759281Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T20:07:47.9792806Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T20:07:47.9826417Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T20:07:47.9860091Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T20:07:47.9891694Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T20:07:47.9928509Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T20:07:47.9965326Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T20:07:48.0014466Z Entering 'third_party/pocketfft' 2024-05-19T20:07:48.0051170Z Entering 'third_party/protobuf' 2024-05-19T20:07:48.0088067Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T20:07:48.0121977Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T20:07:48.0156776Z Entering 'third_party/psimd' 2024-05-19T20:07:48.0192079Z Entering 'third_party/pthreadpool' 2024-05-19T20:07:48.0226140Z Entering 'third_party/pybind11' 2024-05-19T20:07:48.0262581Z Entering 'third_party/python-peachpy' 2024-05-19T20:07:48.0296167Z Entering 'third_party/sleef' 2024-05-19T20:07:48.0327803Z Entering 'third_party/tbb' 2024-05-19T20:07:48.0363861Z Entering 'third_party/tensorpipe' 2024-05-19T20:07:48.0397192Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T20:07:48.0430349Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T20:07:48.0463710Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T20:07:48.0497295Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T20:07:48.0529848Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T20:07:48.0578944Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-05-19T20:07:48.0597017Z http.https://github.com/.extraheader 2024-05-19T20:07:48.0605496Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-05-19T20:07:48.0634180Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-05-19T20:07:48.0840170Z Entering 'android/libs/fbjni' 2024-05-19T20:07:48.0861277Z http.https://github.com/.extraheader 2024-05-19T20:07:48.0884541Z Entering 'third_party/FP16' 2024-05-19T20:07:48.0907748Z http.https://github.com/.extraheader 2024-05-19T20:07:48.0931696Z Entering 'third_party/FXdiv' 2024-05-19T20:07:48.0953502Z http.https://github.com/.extraheader 2024-05-19T20:07:48.0977950Z Entering 'third_party/NNPACK' 2024-05-19T20:07:48.0999033Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1022795Z Entering 'third_party/QNNPACK' 2024-05-19T20:07:48.1046017Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1069007Z Entering 'third_party/VulkanMemoryAllocator' 2024-05-19T20:07:48.1092160Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1115692Z Entering 'third_party/XNNPACK' 2024-05-19T20:07:48.1136778Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1171371Z Entering 'third_party/benchmark' 2024-05-19T20:07:48.1193120Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1216668Z Entering 'third_party/cpp-httplib' 2024-05-19T20:07:48.1238651Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1262549Z Entering 'third_party/cpuinfo' 2024-05-19T20:07:48.1284962Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1307265Z Entering 'third_party/cudnn_frontend' 2024-05-19T20:07:48.1329605Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1352709Z Entering 'third_party/cutlass' 2024-05-19T20:07:48.1376000Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1407544Z Entering 'third_party/eigen' 2024-05-19T20:07:48.1429989Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1454407Z Entering 'third_party/fbgemm' 2024-05-19T20:07:48.1474469Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1498433Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-05-19T20:07:48.1520465Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1542118Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-05-19T20:07:48.1564569Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1587427Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-05-19T20:07:48.1609204Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1636597Z Entering 'third_party/fbgemm/third_party/googletest' 2024-05-19T20:07:48.1657313Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1681355Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-05-19T20:07:48.1702203Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1725928Z Entering 'third_party/flatbuffers' 2024-05-19T20:07:48.1747250Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1771833Z Entering 'third_party/fmt' 2024-05-19T20:07:48.1792889Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1817371Z Entering 'third_party/foxi' 2024-05-19T20:07:48.1839225Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1860371Z Entering 'third_party/gemmlowp/gemmlowp' 2024-05-19T20:07:48.1882158Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1906154Z Entering 'third_party/gloo' 2024-05-19T20:07:48.1927334Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1951890Z Entering 'third_party/googletest' 2024-05-19T20:07:48.1973637Z http.https://github.com/.extraheader 2024-05-19T20:07:48.1996261Z Entering 'third_party/ideep' 2024-05-19T20:07:48.2018398Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2042647Z Entering 'third_party/ideep/mkl-dnn' 2024-05-19T20:07:48.2063525Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2090943Z Entering 'third_party/ios-cmake' 2024-05-19T20:07:48.2111858Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2136291Z Entering 'third_party/ittapi' 2024-05-19T20:07:48.2157826Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2181140Z Entering 'third_party/kineto' 2024-05-19T20:07:48.2203001Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2225151Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-05-19T20:07:48.2247263Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2269615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-05-19T20:07:48.2290860Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2315598Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-05-19T20:07:48.2336804Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2361454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-05-19T20:07:48.2381919Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2406761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-05-19T20:07:48.2426660Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2451127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-05-19T20:07:48.2471997Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2497486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-05-19T20:07:48.2519028Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2542007Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-05-19T20:07:48.2564056Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2587070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-05-19T20:07:48.2608313Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2632874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-05-19T20:07:48.2653760Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2679311Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-05-19T20:07:48.2700150Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2722666Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-05-19T20:07:48.2744127Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2768097Z Entering 'third_party/mimalloc' 2024-05-19T20:07:48.2789465Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2812890Z Entering 'third_party/nccl/nccl' 2024-05-19T20:07:48.2832997Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2857327Z Entering 'third_party/neon2sse' 2024-05-19T20:07:48.2877113Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2898750Z Entering 'third_party/nlohmann' 2024-05-19T20:07:48.2920144Z http.https://github.com/.extraheader 2024-05-19T20:07:48.2943913Z Entering 'third_party/onnx' 2024-05-19T20:07:48.2965921Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3001111Z Entering 'third_party/onnx/third_party/benchmark' 2024-05-19T20:07:48.3022554Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3044410Z Entering 'third_party/onnx/third_party/pybind11' 2024-05-19T20:07:48.3065189Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3090505Z Entering 'third_party/onnx-tensorrt' 2024-05-19T20:07:48.3111791Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3134266Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2024-05-19T20:07:48.3154941Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3181916Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2024-05-19T20:07:48.3203792Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3227829Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2024-05-19T20:07:48.3250112Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3272620Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2024-05-19T20:07:48.3294301Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3322766Z Entering 'third_party/opentelemetry-cpp' 2024-05-19T20:07:48.3344636Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3370190Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-05-19T20:07:48.3391724Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3414999Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-05-19T20:07:48.3435753Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3460010Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-05-19T20:07:48.3480935Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3503514Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-05-19T20:07:48.3525147Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3548939Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-05-19T20:07:48.3570225Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3592482Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-05-19T20:07:48.3614234Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3637061Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-05-19T20:07:48.3658549Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3681659Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-05-19T20:07:48.3702347Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3727421Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-05-19T20:07:48.3748264Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3771568Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-05-19T20:07:48.3791959Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3829282Z Entering 'third_party/pocketfft' 2024-05-19T20:07:48.3850728Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3872436Z Entering 'third_party/protobuf' 2024-05-19T20:07:48.3894569Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3920391Z Entering 'third_party/protobuf/third_party/benchmark' 2024-05-19T20:07:48.3941462Z http.https://github.com/.extraheader 2024-05-19T20:07:48.3963158Z Entering 'third_party/protobuf/third_party/googletest' 2024-05-19T20:07:48.3983843Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4008541Z Entering 'third_party/psimd' 2024-05-19T20:07:48.4029663Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4051916Z Entering 'third_party/pthreadpool' 2024-05-19T20:07:48.4073193Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4096183Z Entering 'third_party/pybind11' 2024-05-19T20:07:48.4117022Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4140396Z Entering 'third_party/python-peachpy' 2024-05-19T20:07:48.4159748Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4181589Z Entering 'third_party/sleef' 2024-05-19T20:07:48.4202594Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4225312Z Entering 'third_party/tbb' 2024-05-19T20:07:48.4245940Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4270971Z Entering 'third_party/tensorpipe' 2024-05-19T20:07:48.4293326Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4316716Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-05-19T20:07:48.4338300Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4361663Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-05-19T20:07:48.4382475Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4405820Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-05-19T20:07:48.4426164Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4449917Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-05-19T20:07:48.4470054Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4491957Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-05-19T20:07:48.4513533Z http.https://github.com/.extraheader 2024-05-19T20:07:48.4762924Z Cleaning up orphan processes