2024-06-26T07:01:47.5836926Z Current runner version: '2.317.0' 2024-06-26T07:01:47.5843125Z Runner name: 'i-0ff3df5957bf06960' 2024-06-26T07:01:47.5843957Z Runner group name: 'Default' 2024-06-26T07:01:47.5844917Z Machine name: 'ip-10-0-70-146' 2024-06-26T07:01:47.5849800Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:01:47.5851747Z Actions: read 2024-06-26T07:01:47.5852278Z Attestations: read 2024-06-26T07:01:47.5852807Z Checks: read 2024-06-26T07:01:47.5853286Z Contents: read 2024-06-26T07:01:47.5853793Z Deployments: read 2024-06-26T07:01:47.5854308Z Discussions: read 2024-06-26T07:01:47.5854850Z Issues: read 2024-06-26T07:01:47.5855317Z Metadata: read 2024-06-26T07:01:47.5855821Z Packages: read 2024-06-26T07:01:47.5856288Z Pages: read 2024-06-26T07:01:47.5856769Z PullRequests: read 2024-06-26T07:01:47.5857372Z RepositoryProjects: read 2024-06-26T07:01:47.5857957Z SecurityEvents: read 2024-06-26T07:01:47.5858462Z Statuses: read 2024-06-26T07:01:47.5858963Z ##[endgroup] 2024-06-26T07:01:47.5862115Z Secret source: Actions 2024-06-26T07:01:47.5862966Z Prepare workflow directory 2024-06-26T07:01:47.9307743Z Prepare all required actions 2024-06-26T07:01:47.9483236Z Getting action download info 2024-06-26T07:01:48.1254844Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:01:48.4343637Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:01:51.1147374Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:01:51.2700538Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:01:51.5433452Z Getting action download info 2024-06-26T07:01:51.6414753Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:01:51.8152312Z Getting action download info 2024-06-26T07:01:51.9144777Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:01:52.0667862Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:01:52.0670181Z ##[group] Inputs 2024-06-26T07:01:52.0670770Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:01:52.0681733Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:01:52.0691059Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:01:52.0692320Z sync-tag: 2024-06-26T07:01:52.0693384Z timeout-minutes: 240 2024-06-26T07:01:52.0693819Z use-gha: 2024-06-26T07:01:52.0694193Z dashboard-tag: 2024-06-26T07:01:52.0694609Z s3-bucket: gha-artifacts 2024-06-26T07:01:52.0695067Z aws-role-to-assume: 2024-06-26T07:01:52.0695501Z ##[endgroup] 2024-06-26T07:01:52.0696449Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:01:52.1297527Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:01:52.1439622Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:01:52.1450345Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:52.1450959Z ##[endgroup] 2024-06-26T07:01:53.0344787Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:01:53.0345370Z with: 2024-06-26T07:01:53.0346102Z github-secret: *** 2024-06-26T07:01:53.0347080Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:01:53.0348091Z activate-with-label: false 2024-06-26T07:01:53.0348482Z label: with-ssh 2024-06-26T07:01:53.0348823Z remove-existing-keys: true 2024-06-26T07:01:53.0349217Z fail-silently: true 2024-06-26T07:01:53.0349559Z env: 2024-06-26T07:01:53.0349870Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:53.0350244Z ##[endgroup] 2024-06-26T07:01:53.1215703Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:01:53.1216747Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:01:53.4312504Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:01:53.5045247Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:01:53.5046461Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:01:53.5678047Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:01:53.5785047Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:01:53.5785624Z with: 2024-06-26T07:01:53.5785935Z submodules: recursive 2024-06-26T07:01:53.5786290Z fetch-depth: 0 2024-06-26T07:01:53.5786607Z env: 2024-06-26T07:01:53.5786903Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:53.5787268Z ##[endgroup] 2024-06-26T07:01:53.5976089Z ##[group]Run retry () { 2024-06-26T07:01:53.5976566Z retry () { 2024-06-26T07:01:53.5977141Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:01:53.5977777Z } 2024-06-26T07:01:53.5978179Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:01:53.5978648Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:01:53.5979190Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:53.5979734Z else 2024-06-26T07:01:53.5980159Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:53.5980631Z fi 2024-06-26T07:01:53.5981096Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:01:53.5989164Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:53.5989741Z env: 2024-06-26T07:01:53.5990100Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:53.5990533Z NO_SUDO: 2024-06-26T07:01:53.5990890Z ##[endgroup] 2024-06-26T07:01:53.6013411Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:55.8631708Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:01:55.8632165Z with: 2024-06-26T07:01:55.8632505Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:55.8632963Z fetch-depth: 0 2024-06-26T07:01:55.8633298Z submodules: recursive 2024-06-26T07:01:55.8633661Z quiet-checkout: true 2024-06-26T07:01:55.8634027Z repository: pytorch/pytorch 2024-06-26T07:01:55.8634528Z token: *** 2024-06-26T07:01:55.8634993Z ssh-strict: true 2024-06-26T07:01:55.8635345Z persist-credentials: true 2024-06-26T07:01:55.8635718Z clean: true 2024-06-26T07:01:55.8636078Z sparse-checkout-cone-mode: true 2024-06-26T07:01:55.8636487Z lfs: false 2024-06-26T07:01:55.8636806Z set-safe-directory: true 2024-06-26T07:01:55.8637165Z env: 2024-06-26T07:01:55.8637879Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:55.8638249Z ##[endgroup] 2024-06-26T07:01:55.9557276Z Syncing repository: pytorch/pytorch 2024-06-26T07:01:55.9559026Z ##[group]Getting Git version info 2024-06-26T07:01:55.9559804Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:55.9560711Z [command]/usr/bin/git version 2024-06-26T07:01:55.9561104Z git version 2.40.1 2024-06-26T07:01:55.9562554Z ##[endgroup] 2024-06-26T07:01:55.9574111Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b6fad071-fd71-43d6-8581-4e15574d5610' before making global git config changes 2024-06-26T07:01:55.9575411Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:01:55.9578262Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:55.9605074Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:55.9609299Z ##[group]Initializing the repository 2024-06-26T07:01:55.9611975Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:55.9636110Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:01:55.9637253Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:01:55.9638447Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:01:55.9639047Z hint: 2024-06-26T07:01:55.9639533Z hint: git config --global init.defaultBranch 2024-06-26T07:01:55.9640036Z hint: 2024-06-26T07:01:55.9640577Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:01:55.9641499Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:01:55.9642443Z hint: 2024-06-26T07:01:55.9642814Z hint: git branch -m 2024-06-26T07:01:55.9643581Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:01:55.9645687Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:01:55.9666364Z ##[endgroup] 2024-06-26T07:01:55.9667009Z ##[group]Disabling automatic garbage collection 2024-06-26T07:01:55.9669180Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:01:55.9688489Z ##[endgroup] 2024-06-26T07:01:55.9689040Z ##[group]Setting up auth 2024-06-26T07:01:55.9694508Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:01:55.9714430Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:01:55.9903382Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:01:55.9924233Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:01:56.0120076Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:01:56.0154005Z ##[endgroup] 2024-06-26T07:01:56.0154602Z ##[group]Fetching the repository 2024-06-26T07:01:56.0160501Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:01:58.1831626Z remote: Enumerating objects: 987392 2024-06-26T07:01:58.1832243Z remote: Enumerating objects: 990209, done. 2024-06-26T07:01:58.1838963Z remote: Counting objects: 0% (1/2817) 2024-06-26T07:01:58.1839525Z remote: Counting objects: 1% (29/2817) 2024-06-26T07:01:58.1840092Z remote: Counting objects: 2% (57/2817) 2024-06-26T07:01:58.1840626Z remote: Counting objects: 3% (85/2817) 2024-06-26T07:01:58.1841153Z remote: Counting objects: 4% (113/2817) 2024-06-26T07:01:58.1841696Z remote: Counting objects: 5% (141/2817) 2024-06-26T07:01:58.1842234Z remote: Counting objects: 6% (170/2817) 2024-06-26T07:01:58.1842775Z remote: Counting objects: 7% (198/2817) 2024-06-26T07:01:58.1843300Z remote: Counting objects: 8% (226/2817) 2024-06-26T07:01:58.1843821Z remote: Counting objects: 9% (254/2817) 2024-06-26T07:01:58.1844355Z remote: Counting objects: 10% (282/2817) 2024-06-26T07:01:58.1844882Z remote: Counting objects: 11% (310/2817) 2024-06-26T07:01:58.1845405Z remote: Counting objects: 12% (339/2817) 2024-06-26T07:01:58.1845920Z remote: Counting objects: 13% (367/2817) 2024-06-26T07:01:58.1846453Z remote: Counting objects: 14% (395/2817) 2024-06-26T07:01:58.1846989Z remote: Counting objects: 15% (423/2817) 2024-06-26T07:01:58.1847511Z remote: Counting objects: 16% (451/2817) 2024-06-26T07:01:58.1848027Z remote: Counting objects: 17% (479/2817) 2024-06-26T07:01:58.1848547Z remote: Counting objects: 18% (508/2817) 2024-06-26T07:01:58.1849068Z remote: Counting objects: 19% (536/2817) 2024-06-26T07:01:58.1849594Z remote: Counting objects: 20% (564/2817) 2024-06-26T07:01:58.1850105Z remote: Counting objects: 21% (592/2817) 2024-06-26T07:01:58.1850629Z remote: Counting objects: 22% (620/2817) 2024-06-26T07:01:58.1851150Z remote: Counting objects: 23% (648/2817) 2024-06-26T07:01:58.1851668Z remote: Counting objects: 24% (677/2817) 2024-06-26T07:01:58.1852181Z remote: Counting objects: 25% (705/2817) 2024-06-26T07:01:58.1852712Z remote: Counting objects: 26% (733/2817) 2024-06-26T07:01:58.1853251Z remote: Counting objects: 27% (761/2817) 2024-06-26T07:01:58.1854114Z remote: Counting objects: 28% (789/2817) 2024-06-26T07:01:58.1854672Z remote: Counting objects: 29% (817/2817) 2024-06-26T07:01:58.1855196Z remote: Counting objects: 30% (846/2817) 2024-06-26T07:01:58.1855716Z remote: Counting objects: 31% (874/2817) 2024-06-26T07:01:58.1856240Z remote: Counting objects: 32% (902/2817) 2024-06-26T07:01:58.1856761Z remote: Counting objects: 33% (930/2817) 2024-06-26T07:01:58.1857285Z remote: Counting objects: 34% (958/2817) 2024-06-26T07:01:58.1857809Z remote: Counting objects: 35% (986/2817) 2024-06-26T07:01:58.1858342Z remote: Counting objects: 36% (1015/2817) 2024-06-26T07:01:58.1858887Z remote: Counting objects: 37% (1043/2817) 2024-06-26T07:01:58.1859438Z remote: Counting objects: 38% (1071/2817) 2024-06-26T07:01:58.1859994Z remote: Counting objects: 39% (1099/2817) 2024-06-26T07:01:58.1860541Z remote: Counting objects: 40% (1127/2817) 2024-06-26T07:01:58.1861082Z remote: Counting objects: 41% (1155/2817) 2024-06-26T07:01:58.1861627Z remote: Counting objects: 42% (1184/2817) 2024-06-26T07:01:58.1862176Z remote: Counting objects: 43% (1212/2817) 2024-06-26T07:01:58.1862716Z remote: Counting objects: 44% (1240/2817) 2024-06-26T07:01:58.1863251Z remote: Counting objects: 45% (1268/2817) 2024-06-26T07:01:58.1864042Z remote: Counting objects: 46% (1296/2817) 2024-06-26T07:01:58.1864580Z remote: Counting objects: 47% (1324/2817) 2024-06-26T07:01:58.1865119Z remote: Counting objects: 48% (1353/2817) 2024-06-26T07:01:58.1865648Z remote: Counting objects: 49% (1381/2817) 2024-06-26T07:01:58.1866186Z remote: Counting objects: 50% (1409/2817) 2024-06-26T07:01:58.1866720Z remote: Counting objects: 51% (1437/2817) 2024-06-26T07:01:58.1867260Z remote: Counting objects: 52% (1465/2817) 2024-06-26T07:01:58.1867802Z remote: Counting objects: 53% (1494/2817) 2024-06-26T07:01:58.1868353Z remote: Counting objects: 54% (1522/2817) 2024-06-26T07:01:58.1868890Z remote: Counting objects: 55% (1550/2817) 2024-06-26T07:01:58.1869431Z remote: Counting objects: 56% (1578/2817) 2024-06-26T07:01:58.1869962Z remote: Counting objects: 57% (1606/2817) 2024-06-26T07:01:58.1870499Z remote: Counting objects: 58% (1634/2817) 2024-06-26T07:01:58.1871040Z remote: Counting objects: 59% (1663/2817) 2024-06-26T07:01:58.1871580Z remote: Counting objects: 60% (1691/2817) 2024-06-26T07:01:58.1872113Z remote: Counting objects: 61% (1719/2817) 2024-06-26T07:01:58.1872652Z remote: Counting objects: 62% (1747/2817) 2024-06-26T07:01:58.1873204Z remote: Counting objects: 63% (1775/2817) 2024-06-26T07:01:58.1873743Z remote: Counting objects: 64% (1803/2817) 2024-06-26T07:01:58.1874281Z remote: Counting objects: 65% (1832/2817) 2024-06-26T07:01:58.1874885Z remote: Counting objects: 66% (1860/2817) 2024-06-26T07:01:58.1875426Z remote: Counting objects: 67% (1888/2817) 2024-06-26T07:01:58.1875965Z remote: Counting objects: 68% (1916/2817) 2024-06-26T07:01:58.1876505Z remote: Counting objects: 69% (1944/2817) 2024-06-26T07:01:58.1877249Z remote: Counting objects: 70% (1972/2817) 2024-06-26T07:01:58.1878021Z remote: Counting objects: 71% (2001/2817) 2024-06-26T07:01:58.1880350Z remote: Counting objects: 72% (2029/2817) 2024-06-26T07:01:58.1882884Z remote: Counting objects: 73% (2057/2817) 2024-06-26T07:01:58.1885578Z remote: Counting objects: 74% (2085/2817) 2024-06-26T07:01:58.1886120Z remote: Counting objects: 75% (2113/2817) 2024-06-26T07:01:58.1886658Z remote: Counting objects: 76% (2141/2817) 2024-06-26T07:01:58.1887652Z remote: Counting objects: 77% (2170/2817) 2024-06-26T07:01:58.1888190Z remote: Counting objects: 78% (2198/2817) 2024-06-26T07:01:58.1888836Z remote: Counting objects: 79% (2226/2817) 2024-06-26T07:01:58.1889382Z remote: Counting objects: 80% (2254/2817) 2024-06-26T07:01:58.1889922Z remote: Counting objects: 81% (2282/2817) 2024-06-26T07:01:58.1890461Z remote: Counting objects: 82% (2310/2817) 2024-06-26T07:01:58.1890993Z remote: Counting objects: 83% (2339/2817) 2024-06-26T07:01:58.1891536Z remote: Counting objects: 84% (2367/2817) 2024-06-26T07:01:58.1892074Z remote: Counting objects: 85% (2395/2817) 2024-06-26T07:01:58.1892621Z remote: Counting objects: 86% (2423/2817) 2024-06-26T07:01:58.1893153Z remote: Counting objects: 87% (2451/2817) 2024-06-26T07:01:58.1893693Z remote: Counting objects: 88% (2479/2817) 2024-06-26T07:01:58.1894235Z remote: Counting objects: 89% (2508/2817) 2024-06-26T07:01:58.1894772Z remote: Counting objects: 90% (2536/2817) 2024-06-26T07:01:58.1895304Z remote: Counting objects: 91% (2564/2817) 2024-06-26T07:01:58.1895849Z remote: Counting objects: 92% (2592/2817) 2024-06-26T07:01:58.1896387Z remote: Counting objects: 93% (2620/2817) 2024-06-26T07:01:58.1896923Z remote: Counting objects: 94% (2648/2817) 2024-06-26T07:01:58.1897452Z remote: Counting objects: 95% (2677/2817) 2024-06-26T07:01:58.1897990Z remote: Counting objects: 96% (2705/2817) 2024-06-26T07:01:58.1898606Z remote: Counting objects: 97% (2733/2817) 2024-06-26T07:01:58.1899144Z remote: Counting objects: 98% (2761/2817) 2024-06-26T07:01:58.1899674Z remote: Counting objects: 99% (2789/2817) 2024-06-26T07:01:58.1900211Z remote: Counting objects: 100% (2817/2817) 2024-06-26T07:01:58.1900790Z remote: Counting objects: 100% (2817/2817), done. 2024-06-26T07:01:58.2064969Z remote: Compressing objects: 0% (1/1140) 2024-06-26T07:01:58.2637691Z remote: Compressing objects: 1% (12/1140) 2024-06-26T07:01:58.3055442Z remote: Compressing objects: 2% (23/1140) 2024-06-26T07:01:58.3378250Z remote: Compressing objects: 3% (35/1140) 2024-06-26T07:01:58.3569990Z remote: Compressing objects: 4% (46/1140) 2024-06-26T07:01:58.3875881Z remote: Compressing objects: 5% (57/1140) 2024-06-26T07:01:58.4105372Z remote: Compressing objects: 6% (69/1140) 2024-06-26T07:01:58.5057112Z remote: Compressing objects: 7% (80/1140) 2024-06-26T07:01:58.5919332Z remote: Compressing objects: 8% (92/1140) 2024-06-26T07:01:58.6568479Z remote: Compressing objects: 9% (103/1140) 2024-06-26T07:01:58.6795320Z remote: Compressing objects: 10% (114/1140) 2024-06-26T07:01:58.6927617Z remote: Compressing objects: 11% (126/1140) 2024-06-26T07:01:58.7024448Z remote: Compressing objects: 12% (137/1140) 2024-06-26T07:01:58.7080577Z remote: Compressing objects: 13% (149/1140) 2024-06-26T07:01:58.7081710Z remote: Compressing objects: 14% (160/1140) 2024-06-26T07:01:58.7162221Z remote: Compressing objects: 15% (171/1140) 2024-06-26T07:01:58.7162790Z remote: Compressing objects: 16% (183/1140) 2024-06-26T07:01:58.7169475Z remote: Compressing objects: 17% (194/1140) 2024-06-26T07:01:58.7197247Z remote: Compressing objects: 18% (206/1140) 2024-06-26T07:01:58.7228363Z remote: Compressing objects: 19% (217/1140) 2024-06-26T07:01:58.7257197Z remote: Compressing objects: 20% (228/1140) 2024-06-26T07:01:58.7284118Z remote: Compressing objects: 21% (240/1140) 2024-06-26T07:01:58.7313409Z remote: Compressing objects: 22% (251/1140) 2024-06-26T07:01:58.7323647Z remote: Compressing objects: 23% (263/1140) 2024-06-26T07:01:58.7331980Z remote: Compressing objects: 24% (274/1140) 2024-06-26T07:01:58.7343253Z remote: Compressing objects: 25% (285/1140) 2024-06-26T07:01:58.7349505Z remote: Compressing objects: 26% (297/1140) 2024-06-26T07:01:58.7355595Z remote: Compressing objects: 27% (308/1140) 2024-06-26T07:01:58.7369822Z remote: Compressing objects: 28% (320/1140) 2024-06-26T07:01:58.7377432Z remote: Compressing objects: 29% (331/1140) 2024-06-26T07:01:58.7396201Z remote: Compressing objects: 30% (342/1140) 2024-06-26T07:01:58.7402970Z remote: Compressing objects: 31% (354/1140) 2024-06-26T07:01:58.7414475Z remote: Compressing objects: 32% (365/1140) 2024-06-26T07:01:58.7425265Z remote: Compressing objects: 33% (377/1140) 2024-06-26T07:01:58.7442426Z remote: Compressing objects: 34% (388/1140) 2024-06-26T07:01:58.7457965Z remote: Compressing objects: 35% (399/1140) 2024-06-26T07:01:58.7482943Z remote: Compressing objects: 36% (411/1140) 2024-06-26T07:01:58.7504140Z remote: Compressing objects: 37% (422/1140) 2024-06-26T07:01:58.7520458Z remote: Compressing objects: 38% (434/1140) 2024-06-26T07:01:58.7533033Z remote: Compressing objects: 39% (445/1140) 2024-06-26T07:01:58.7556745Z remote: Compressing objects: 40% (456/1140) 2024-06-26T07:01:58.7588665Z remote: Compressing objects: 41% (468/1140) 2024-06-26T07:01:58.7608337Z remote: Compressing objects: 42% (479/1140) 2024-06-26T07:01:58.7620338Z remote: Compressing objects: 43% (491/1140) 2024-06-26T07:01:58.7627611Z remote: Compressing objects: 44% (502/1140) 2024-06-26T07:01:58.7633036Z remote: Compressing objects: 45% (513/1140) 2024-06-26T07:01:58.7643576Z remote: Compressing objects: 46% (525/1140) 2024-06-26T07:01:58.7665013Z remote: Compressing objects: 47% (536/1140) 2024-06-26T07:01:58.7673039Z remote: Compressing objects: 48% (548/1140) 2024-06-26T07:01:58.7681853Z remote: Compressing objects: 49% (559/1140) 2024-06-26T07:01:58.7684418Z remote: Compressing objects: 50% (570/1140) 2024-06-26T07:01:58.7700898Z remote: Compressing objects: 51% (582/1140) 2024-06-26T07:01:58.7726426Z remote: Compressing objects: 52% (593/1140) 2024-06-26T07:01:58.7750966Z remote: Compressing objects: 53% (605/1140) 2024-06-26T07:01:58.7764374Z remote: Compressing objects: 54% (616/1140) 2024-06-26T07:01:58.7776756Z remote: Compressing objects: 55% (627/1140) 2024-06-26T07:01:58.7783238Z remote: Compressing objects: 56% (639/1140) 2024-06-26T07:01:58.7794086Z remote: Compressing objects: 57% (650/1140) 2024-06-26T07:01:58.7805457Z remote: Compressing objects: 58% (662/1140) 2024-06-26T07:01:58.7817447Z remote: Compressing objects: 59% (673/1140) 2024-06-26T07:01:58.7828037Z remote: Compressing objects: 60% (684/1140) 2024-06-26T07:01:58.7836116Z remote: Compressing objects: 61% (696/1140) 2024-06-26T07:01:58.7849273Z remote: Compressing objects: 62% (707/1140) 2024-06-26T07:01:58.7859911Z remote: Compressing objects: 63% (719/1140) 2024-06-26T07:01:58.7875549Z remote: Compressing objects: 64% (730/1140) 2024-06-26T07:01:58.7878398Z remote: Compressing objects: 65% (741/1140) 2024-06-26T07:01:58.7886744Z remote: Compressing objects: 66% (753/1140) 2024-06-26T07:01:58.7894333Z remote: Compressing objects: 67% (764/1140) 2024-06-26T07:01:58.7899819Z remote: Compressing objects: 68% (776/1140) 2024-06-26T07:01:58.7905546Z remote: Compressing objects: 69% (787/1140) 2024-06-26T07:01:58.7911681Z remote: Compressing objects: 70% (798/1140) 2024-06-26T07:01:58.7912520Z remote: Compressing objects: 71% (810/1140) 2024-06-26T07:01:58.7916655Z remote: Compressing objects: 72% (821/1140) 2024-06-26T07:01:58.7918956Z remote: Compressing objects: 73% (833/1140) 2024-06-26T07:01:58.7920118Z remote: Compressing objects: 74% (844/1140) 2024-06-26T07:01:58.7921549Z remote: Compressing objects: 75% (855/1140) 2024-06-26T07:01:58.7923272Z remote: Compressing objects: 76% (867/1140) 2024-06-26T07:01:58.7924294Z remote: Compressing objects: 77% (878/1140) 2024-06-26T07:01:58.7924865Z remote: Compressing objects: 78% (890/1140) 2024-06-26T07:01:58.7925424Z remote: Compressing objects: 79% (901/1140) 2024-06-26T07:01:58.7927781Z remote: Compressing objects: 80% (912/1140) 2024-06-26T07:01:58.7929843Z remote: Compressing objects: 81% (924/1140) 2024-06-26T07:01:58.7931888Z remote: Compressing objects: 82% (935/1140) 2024-06-26T07:01:58.7934775Z remote: Compressing objects: 83% (947/1140) 2024-06-26T07:01:58.7936894Z remote: Compressing objects: 84% (958/1140) 2024-06-26T07:01:58.7939651Z remote: Compressing objects: 85% (969/1140) 2024-06-26T07:01:58.7942054Z remote: Compressing objects: 86% (981/1140) 2024-06-26T07:01:58.7942622Z remote: Compressing objects: 87% (992/1140) 2024-06-26T07:01:58.7945439Z remote: Compressing objects: 88% (1004/1140) 2024-06-26T07:01:58.7946019Z remote: Compressing objects: 89% (1015/1140) 2024-06-26T07:01:58.7952096Z remote: Compressing objects: 90% (1026/1140) 2024-06-26T07:01:58.7952676Z remote: Compressing objects: 91% (1038/1140) 2024-06-26T07:01:58.7953244Z remote: Compressing objects: 92% (1049/1140) 2024-06-26T07:01:58.7953818Z remote: Compressing objects: 93% (1061/1140) 2024-06-26T07:01:58.7954393Z remote: Compressing objects: 94% (1072/1140) 2024-06-26T07:01:58.7955229Z remote: Compressing objects: 95% (1083/1140) 2024-06-26T07:01:58.7960602Z remote: Compressing objects: 96% (1095/1140) 2024-06-26T07:01:58.7961199Z remote: Compressing objects: 97% (1106/1140) 2024-06-26T07:01:58.7961948Z remote: Compressing objects: 98% (1118/1140) 2024-06-26T07:01:58.7962689Z remote: Compressing objects: 99% (1129/1140) 2024-06-26T07:01:58.7963452Z remote: Compressing objects: 100% (1140/1140) 2024-06-26T07:01:58.7964257Z remote: Compressing objects: 100% (1140/1140), done. 2024-06-26T07:02:17.7476508Z remote: Total 990209 (delta 2280), reused 2016 (delta 1676), pack-reused 987392 2024-06-26T07:02:43.9502145Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:02:43.9524869Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:43.9529319Z ##[endgroup] 2024-06-26T07:02:43.9529908Z ##[group]Determining the checkout info 2024-06-26T07:02:43.9531664Z ##[endgroup] 2024-06-26T07:02:43.9532318Z ##[group]Checking out the ref 2024-06-26T07:02:43.9534243Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:45.1811650Z ##[endgroup] 2024-06-26T07:02:45.1812287Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:02:45.1817016Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:02:45.1856070Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:02:45.1880902Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:02:45.1900920Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:02:45.1918009Z ##[endgroup] 2024-06-26T07:02:45.1918597Z ##[group]Fetching submodules 2024-06-26T07:02:45.1921625Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:02:45.2130879Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:02:45.2334850Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:02:45.2336343Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:02:45.2337828Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:02:45.2339443Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:02:45.2341168Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:45.2342830Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:02:45.2344738Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:02:45.2346871Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:02:45.2349260Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:02:45.2351775Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:02:45.2353790Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:02:45.2356421Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:02:45.2359236Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:02:45.2362055Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:02:45.2364467Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:02:45.2367117Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:02:45.2370020Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:45.2372647Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:02:45.2375594Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:02:45.2378331Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:02:45.2381399Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:02:45.2384448Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:02:45.2387714Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:02:45.2390820Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:02:45.2394084Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:02:45.2397731Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:02:45.2401770Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:02:45.2404854Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:02:45.2408520Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:02:45.2411974Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:02:45.2415841Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:02:45.2419355Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:02:45.2423140Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:02:45.2426765Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:02:45.2430808Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:02:45.2451334Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:02:45.4881488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:02:45.6802792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:02:45.8421953Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:02:46.0772034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:02:48.1481619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:02:57.8661781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:02:58.2368407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:02:58.7191591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:02:59.2912060Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:03:00.4053022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:03:02.1167220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:03:08.0728436Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:03:09.3314667Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:03:11.1102506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:03:12.3470457Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:03:12.5113245Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:03:12.9028119Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:03:13.2159688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:03:14.1915682Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:03:14.5291490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:03:14.7889133Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:03:16.2715202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:03:16.9825182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:03:17.6115507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:03:23.1973756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:03:25.6214074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:03:31.2895306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:03:31.4915058Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:03:39.7308488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:03:39.9173693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:03:40.1059272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:03:41.1180039Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:03:41.4371797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:03:42.0304287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:03:42.4461643Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:03:42.4547704Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:03:42.4613980Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:03:42.4794579Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:03:42.5108047Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:03:43.2785637Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:03:43.2966088Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:03:43.3307025Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:03:43.4160336Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:03:43.4426091Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:03:43.8702763Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:03:44.0825498Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:03:44.1464336Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:03:44.1474653Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:44.1476523Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:44.1478530Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:44.1480511Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:44.1482464Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:44.1500157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:03:45.1299892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:03:45.7163912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:03:47.4280389Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:03:48.4629103Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:03:48.7933145Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:03:48.8773959Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:03:49.2203255Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:03:49.2742047Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:03:49.2826220Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:03:49.3730186Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:03:49.4018687Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:03:49.4086658Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:03:49.4391679Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:03:49.4593213Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:03:49.4977547Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:03:49.5071238Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:03:49.5084185Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:49.5100248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:04:01.6693984Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:04:01.6831390Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:04:01.7610857Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:04:01.7625924Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:01.7628100Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:01.7630161Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:01.7647707Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:04:02.2729381Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:04:03.4938743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:04:04.5436472Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:04:04.5446744Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:04.5448532Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:04.5450274Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:04.5452361Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:04.5454176Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:04.5456328Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:04.5458347Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:04.5460350Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:04.5479148Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:04:05.5429804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:04:05.9100724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:04:07.0743214Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:04:07.3755592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:04:07.8466706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:04:08.8048924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:04:14.3438064Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:04:14.6915214Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:04:14.7051768Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:04:14.7345144Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:04:14.7441495Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:04:14.7452136Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:14.7469044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:04:15.0612642Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:04:15.0749927Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:04:15.1086761Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:04:15.1898186Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:04:15.2017718Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:04:15.2306897Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:04:15.2798748Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:04:15.3105858Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:04:15.3304259Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:04:15.4165426Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:04:15.6884889Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:04:15.6913358Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:15.6915310Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:15.6933841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:04:16.1142399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:04:17.2403003Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:04:17.2677682Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:04:17.3201841Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:04:17.3214127Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:17.3221520Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:17.3223222Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:17.3251169Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:17.3253523Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:17.3255609Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:17.3258031Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:17.3259709Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:17.3261160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:04:17.7909153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:04:18.7679306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:04:19.0777901Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:04:25.4899776Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:04:25.7554704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:04:25.9533280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:04:26.2411006Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:04:33.5168246Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:04:33.5498889Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:04:33.5623901Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:04:33.6461085Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:04:33.6553906Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:04:33.6662570Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:04:33.6778545Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:04:33.6788375Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:33.6790246Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:33.6807513Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:04:35.4915274Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:04:36.6505772Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:04:36.6874805Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:04:37.0451800Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:04:37.0534103Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:04:37.2692040Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:04:37.2705376Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:37.2707373Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:37.2726955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:04:37.6293561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:04:38.5985730Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:04:38.6607016Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:04:38.6666996Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:04:38.6755933Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:04:38.7034457Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:04:38.7259298Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:04:38.7589309Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:04:38.7792049Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:04:38.7803273Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:38.7804911Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:38.7806928Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:38.7808864Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:38.7826391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:04:39.7616652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:04:39.9672166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:04:40.9595732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:04:41.9933820Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:04:42.0048332Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:04:42.0616122Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:04:42.0844746Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:04:42.0854196Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:42.0871329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:04:42.3156916Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:04:42.3180513Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:04:42.3383045Z Entering 'android/libs/fbjni' 2024-06-26T07:04:42.3409562Z Entering 'third_party/FP16' 2024-06-26T07:04:42.3434858Z Entering 'third_party/FXdiv' 2024-06-26T07:04:42.3461205Z Entering 'third_party/NNPACK' 2024-06-26T07:04:42.3490554Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:42.3516121Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:42.3553143Z Entering 'third_party/benchmark' 2024-06-26T07:04:42.3578080Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:42.3606841Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:42.3632797Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:42.3661391Z Entering 'third_party/cutlass' 2024-06-26T07:04:42.3693523Z Entering 'third_party/eigen' 2024-06-26T07:04:42.3724183Z Entering 'third_party/fbgemm' 2024-06-26T07:04:42.3749373Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:42.3777706Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:42.3804411Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:42.3834444Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:42.3861015Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:42.3889573Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:42.3919459Z Entering 'third_party/fmt' 2024-06-26T07:04:42.3947931Z Entering 'third_party/foxi' 2024-06-26T07:04:42.3975313Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:42.4001709Z Entering 'third_party/gloo' 2024-06-26T07:04:42.4032361Z Entering 'third_party/googletest' 2024-06-26T07:04:42.4060704Z Entering 'third_party/ideep' 2024-06-26T07:04:42.4087438Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:42.4121093Z Entering 'third_party/ittapi' 2024-06-26T07:04:42.4149694Z Entering 'third_party/kineto' 2024-06-26T07:04:42.4177209Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:42.4205730Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:42.4234059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:42.4262296Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:42.4291227Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:42.4316042Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:42.4344182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:42.4371807Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:42.4401562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:42.4428065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:42.4455796Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:42.4482857Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:42.4509873Z Entering 'third_party/mimalloc' 2024-06-26T07:04:42.4537439Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:42.4563451Z Entering 'third_party/nlohmann' 2024-06-26T07:04:42.4592981Z Entering 'third_party/onnx' 2024-06-26T07:04:42.4634803Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:42.4664600Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:42.4694242Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:42.4726076Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:42.4753054Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:42.4782219Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:42.4807762Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:42.4833811Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:42.4861995Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:42.4887393Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:42.4913137Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:42.4942692Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:42.4972101Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:42.5013700Z Entering 'third_party/pocketfft' 2024-06-26T07:04:42.5041879Z Entering 'third_party/protobuf' 2024-06-26T07:04:42.5071438Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:42.5097212Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:42.5125387Z Entering 'third_party/psimd' 2024-06-26T07:04:42.5151571Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:42.5178669Z Entering 'third_party/pybind11' 2024-06-26T07:04:42.5206956Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:42.5235310Z Entering 'third_party/sleef' 2024-06-26T07:04:42.5264048Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:42.5292567Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:42.5320610Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:42.5346810Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:42.5374457Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:42.5399369Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:42.5439940Z ##[endgroup] 2024-06-26T07:04:42.5440949Z ##[group]Persisting credentials for submodules 2024-06-26T07:04:42.5444926Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:04:42.5654794Z Entering 'android/libs/fbjni' 2024-06-26T07:04:42.5690700Z Entering 'third_party/FP16' 2024-06-26T07:04:42.5726287Z Entering 'third_party/FXdiv' 2024-06-26T07:04:42.5761734Z Entering 'third_party/NNPACK' 2024-06-26T07:04:42.5795213Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:42.5831694Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:42.5877007Z Entering 'third_party/benchmark' 2024-06-26T07:04:42.5914336Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:42.5949713Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:42.5985381Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:42.6021269Z Entering 'third_party/cutlass' 2024-06-26T07:04:42.6060353Z Entering 'third_party/eigen' 2024-06-26T07:04:42.6095849Z Entering 'third_party/fbgemm' 2024-06-26T07:04:42.6127641Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:42.6163545Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:42.6196635Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:42.6234448Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:42.6269717Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:42.6305446Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:42.6343460Z Entering 'third_party/fmt' 2024-06-26T07:04:42.6379359Z Entering 'third_party/foxi' 2024-06-26T07:04:42.6414561Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:42.6446384Z Entering 'third_party/gloo' 2024-06-26T07:04:42.6481193Z Entering 'third_party/googletest' 2024-06-26T07:04:42.6513990Z Entering 'third_party/ideep' 2024-06-26T07:04:42.6549410Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:42.6588682Z Entering 'third_party/ittapi' 2024-06-26T07:04:42.6624426Z Entering 'third_party/kineto' 2024-06-26T07:04:42.6656147Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:42.6689920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:42.6726264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:42.6760376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:42.6793955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:42.6827619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:42.6864246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:42.6899028Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:42.6934073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:42.6968226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:42.7005099Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:42.7036673Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:42.7071871Z Entering 'third_party/mimalloc' 2024-06-26T07:04:42.7107353Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:42.7143585Z Entering 'third_party/nlohmann' 2024-06-26T07:04:42.7180663Z Entering 'third_party/onnx' 2024-06-26T07:04:42.7225374Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:42.7260675Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:42.7296999Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:42.7333686Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:42.7367956Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:42.7401834Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:42.7434924Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:42.7470914Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:42.7505611Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:42.7540233Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:42.7574216Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:42.7610384Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:42.7646338Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:42.7698629Z Entering 'third_party/pocketfft' 2024-06-26T07:04:42.7734424Z Entering 'third_party/protobuf' 2024-06-26T07:04:42.7769659Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:42.7804059Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:42.7840086Z Entering 'third_party/psimd' 2024-06-26T07:04:42.7875880Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:42.7910335Z Entering 'third_party/pybind11' 2024-06-26T07:04:42.7946047Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:42.7981609Z Entering 'third_party/sleef' 2024-06-26T07:04:42.8014592Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:42.8048069Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:42.8081378Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:42.8114524Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:42.8149546Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:42.8180608Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:42.8227721Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:04:42.8436197Z Entering 'android/libs/fbjni' 2024-06-26T07:04:42.8470410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:04:42.8483222Z Entering 'third_party/FP16' 2024-06-26T07:04:42.8513961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:04:42.8526459Z Entering 'third_party/FXdiv' 2024-06-26T07:04:42.8558569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:04:42.8570666Z Entering 'third_party/NNPACK' 2024-06-26T07:04:42.8600949Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:04:42.8612937Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:42.8645587Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:04:42.8658625Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:42.8691351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:04:42.8715809Z Entering 'third_party/benchmark' 2024-06-26T07:04:42.8747094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:42.8761149Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:42.8792397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:04:42.8805858Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:42.8835273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:42.8847320Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:42.8878101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:04:42.8890138Z Entering 'third_party/cutlass' 2024-06-26T07:04:42.8920672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:42.8939532Z Entering 'third_party/eigen' 2024-06-26T07:04:42.8971255Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:04:42.8986308Z Entering 'third_party/fbgemm' 2024-06-26T07:04:42.9020430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:04:42.9031906Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:42.9064708Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:04:42.9077735Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:42.9110127Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:42.9123314Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:42.9154225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:42.9172814Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:42.9205261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:42.9217705Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:42.9249611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:04:42.9264330Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:42.9296280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:04:42.9311846Z Entering 'third_party/fmt' 2024-06-26T07:04:42.9344807Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:42.9356830Z Entering 'third_party/foxi' 2024-06-26T07:04:42.9387646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:04:42.9400047Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:42.9432920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:04:42.9446411Z Entering 'third_party/gloo' 2024-06-26T07:04:42.9478418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:04:42.9490822Z Entering 'third_party/googletest' 2024-06-26T07:04:42.9522510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:42.9535487Z Entering 'third_party/ideep' 2024-06-26T07:04:42.9566258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:04:42.9578252Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:42.9610331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:04:42.9629037Z Entering 'third_party/ittapi' 2024-06-26T07:04:42.9662296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:04:42.9675274Z Entering 'third_party/kineto' 2024-06-26T07:04:42.9706903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:04:42.9720187Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:42.9752134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:04:42.9764182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:42.9796244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:04:42.9811092Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:42.9843093Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:04:42.9856063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:42.9889918Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:42.9901888Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:42.9934551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:04:42.9945759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:42.9977557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:04:42.9990838Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:43.0023115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:04:43.0036492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:43.0069888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:43.0082629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:43.0114068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:04:43.0127196Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:43.0158795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:04:43.0174405Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:43.0205552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:04:43.0218332Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:43.0250520Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:04:43.0263735Z Entering 'third_party/mimalloc' 2024-06-26T07:04:43.0296234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:04:43.0307795Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:43.0339354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:04:43.0351350Z Entering 'third_party/nlohmann' 2024-06-26T07:04:43.0384550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:04:43.0398976Z Entering 'third_party/onnx' 2024-06-26T07:04:43.0429237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:04:43.0454477Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:43.0485544Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:43.0499135Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:43.0530166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:43.0543018Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:43.0576046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:04:43.0589742Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:43.0620380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:43.0633128Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:43.0665585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:43.0678072Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:43.0708502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:04:43.0721027Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:43.0751946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:04:43.0764528Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:43.0795395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:04:43.0807969Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:43.0840220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:04:43.0852189Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:43.0884079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:04:43.0895651Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:43.0926353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:04:43.0939269Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:43.0970270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:04:43.0984526Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:43.1014644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:04:43.1043739Z Entering 'third_party/pocketfft' 2024-06-26T07:04:43.1075905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:04:43.1089574Z Entering 'third_party/protobuf' 2024-06-26T07:04:43.1121953Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:04:43.1137389Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:43.1168087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:43.1181827Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:43.1211994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:43.1225905Z Entering 'third_party/psimd' 2024-06-26T07:04:43.1258614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:04:43.1271514Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:43.1303962Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:04:43.1316658Z Entering 'third_party/pybind11' 2024-06-26T07:04:43.1349399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:43.1363420Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:43.1394271Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:04:43.1407663Z Entering 'third_party/sleef' 2024-06-26T07:04:43.1438542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:04:43.1451009Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:43.1483032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:04:43.1495413Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:43.1527401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:43.1539330Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:43.1570147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:04:43.1583407Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:43.1614994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:04:43.1627869Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:43.1660613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:43.1673110Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:43.1706571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:04:43.2526270Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:04:43.2736871Z Entering 'android/libs/fbjni' 2024-06-26T07:04:43.2765221Z Entering 'third_party/FP16' 2024-06-26T07:04:43.2795345Z Entering 'third_party/FXdiv' 2024-06-26T07:04:43.2824185Z Entering 'third_party/NNPACK' 2024-06-26T07:04:43.2854609Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:43.2884561Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:43.2923898Z Entering 'third_party/benchmark' 2024-06-26T07:04:43.2950562Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:43.2978301Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:43.3006161Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:43.3033572Z Entering 'third_party/cutlass' 2024-06-26T07:04:43.3067880Z Entering 'third_party/eigen' 2024-06-26T07:04:43.3098889Z Entering 'third_party/fbgemm' 2024-06-26T07:04:43.3126437Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:43.3153273Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:43.3181991Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:43.3213274Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:43.3240667Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:43.3269184Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:43.3298701Z Entering 'third_party/fmt' 2024-06-26T07:04:43.3326425Z Entering 'third_party/foxi' 2024-06-26T07:04:43.3354792Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:43.3383581Z Entering 'third_party/gloo' 2024-06-26T07:04:43.3412085Z Entering 'third_party/googletest' 2024-06-26T07:04:43.3442021Z Entering 'third_party/ideep' 2024-06-26T07:04:43.3470551Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:43.3504250Z Entering 'third_party/ittapi' 2024-06-26T07:04:43.3534374Z Entering 'third_party/kineto' 2024-06-26T07:04:43.3563584Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:43.3592425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:43.3621898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:43.3649989Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:43.3677075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:43.3704527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:43.3734679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:43.3763263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:43.3791733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:43.3819304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:43.3848838Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:43.3876356Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:43.3906094Z Entering 'third_party/mimalloc' 2024-06-26T07:04:43.3932448Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:43.3959557Z Entering 'third_party/nlohmann' 2024-06-26T07:04:43.3989783Z Entering 'third_party/onnx' 2024-06-26T07:04:43.4030085Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:43.4060101Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:43.4087133Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:43.4116045Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:43.4145515Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:43.4174272Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:43.4201756Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:43.4228418Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:43.4257306Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:43.4283733Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:43.4308934Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:43.4337834Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:43.4367878Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:43.4410009Z Entering 'third_party/pocketfft' 2024-06-26T07:04:43.4438761Z Entering 'third_party/protobuf' 2024-06-26T07:04:43.4469615Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:43.4496029Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:43.4523004Z Entering 'third_party/psimd' 2024-06-26T07:04:43.4550830Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:43.4579776Z Entering 'third_party/pybind11' 2024-06-26T07:04:43.4606340Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:43.4632134Z Entering 'third_party/sleef' 2024-06-26T07:04:43.4660361Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:43.4686215Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:43.4713878Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:43.4742386Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:43.4768122Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:43.4793217Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:43.4835755Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:04:43.5044546Z Entering 'android/libs/fbjni' 2024-06-26T07:04:43.5070899Z Entering 'third_party/FP16' 2024-06-26T07:04:43.5100308Z Entering 'third_party/FXdiv' 2024-06-26T07:04:43.5127813Z Entering 'third_party/NNPACK' 2024-06-26T07:04:43.5157725Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:43.5185414Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:43.5226134Z Entering 'third_party/benchmark' 2024-06-26T07:04:43.5253515Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:43.5281016Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:43.5307006Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:43.5334993Z Entering 'third_party/cutlass' 2024-06-26T07:04:43.5367324Z Entering 'third_party/eigen' 2024-06-26T07:04:43.5397869Z Entering 'third_party/fbgemm' 2024-06-26T07:04:43.5425611Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:43.5454321Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:43.5481006Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:43.5513043Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:43.5541563Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:43.5569292Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:43.5598716Z Entering 'third_party/fmt' 2024-06-26T07:04:43.5626918Z Entering 'third_party/foxi' 2024-06-26T07:04:43.5655815Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:43.5685103Z Entering 'third_party/gloo' 2024-06-26T07:04:43.5712493Z Entering 'third_party/googletest' 2024-06-26T07:04:43.5741762Z Entering 'third_party/ideep' 2024-06-26T07:04:43.5767618Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:43.5801122Z Entering 'third_party/ittapi' 2024-06-26T07:04:43.5829591Z Entering 'third_party/kineto' 2024-06-26T07:04:43.5857218Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:43.5885931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:43.5915984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:43.5943195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:43.5971018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:43.5999262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:43.6027819Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:43.6054881Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:43.6082805Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:43.6112246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:43.6140593Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:43.6168367Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:43.6197796Z Entering 'third_party/mimalloc' 2024-06-26T07:04:43.6226441Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:43.6255069Z Entering 'third_party/nlohmann' 2024-06-26T07:04:43.6283866Z Entering 'third_party/onnx' 2024-06-26T07:04:43.6324391Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:43.6354299Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:43.6386261Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:43.6414933Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:43.6442937Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:43.6469193Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:43.6495908Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:43.6525625Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:43.6551887Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:43.6580249Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:43.6605293Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:43.6635318Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:43.6662820Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:43.6703349Z Entering 'third_party/pocketfft' 2024-06-26T07:04:43.6732287Z Entering 'third_party/protobuf' 2024-06-26T07:04:43.6763078Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:43.6789875Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:43.6819658Z Entering 'third_party/psimd' 2024-06-26T07:04:43.6845093Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:43.6874137Z Entering 'third_party/pybind11' 2024-06-26T07:04:43.6904259Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:43.6932066Z Entering 'third_party/sleef' 2024-06-26T07:04:43.6960474Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:43.6987372Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:43.7013475Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:43.7041170Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:43.7067429Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:43.7094931Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:43.7132663Z ##[endgroup] 2024-06-26T07:04:43.7160985Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:04:43.7182459Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:04:43.7356105Z Prepare all required actions 2024-06-26T07:04:43.7356658Z Getting action download info 2024-06-26T07:04:43.8788601Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:04:43.8789127Z env: 2024-06-26T07:04:43.8789506Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.8789948Z ##[endgroup] 2024-06-26T07:04:43.8851327Z ##[group]Run set -euo pipefail 2024-06-26T07:04:43.8851861Z set -euo pipefail 2024-06-26T07:04:43.8852349Z function get_ec2_metadata() { 2024-06-26T07:04:43.8852981Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:04:43.8853960Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:04:43.8854819Z  category=$1 2024-06-26T07:04:43.8855450Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:04:43.8856183Z  runner_name_str=i-0ff3df5957bf06960 2024-06-26T07:04:43.8856779Z  if [[ -f /.inarc ]]; then 2024-06-26T07:04:43.8857369Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:04:43.8858036Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:04:43.8858820Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:04:43.8859520Z  else 2024-06-26T07:04:43.8860205Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:04:43.8861036Z  fi 2024-06-26T07:04:43.8861516Z } 2024-06-26T07:04:43.8862087Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:04:43.8862852Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:04:43.8863603Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:04:43.8864271Z echo "system info $(uname -a)" 2024-06-26T07:04:43.8872888Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.8873475Z env: 2024-06-26T07:04:43.8873855Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.8874302Z ##[endgroup] 2024-06-26T07:04:43.8946278Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:04:43.8988817Z instance-id: i-0ff3df5957bf06960 2024-06-26T07:04:43.9031074Z instance-type: g5.4xlarge 2024-06-26T07:04:43.9037014Z system info Linux ip-10-0-70-146.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:04:43.9057615Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:43.9058545Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:43.9066137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.9066779Z env: 2024-06-26T07:04:43.9067090Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.9067465Z ##[endgroup] 2024-06-26T07:04:43.9129880Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:04:43.9130518Z if systemctl is-active --quiet docker; then 2024-06-26T07:04:43.9131115Z  echo "Docker daemon is running..."; 2024-06-26T07:04:43.9131616Z else 2024-06-26T07:04:43.9132156Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:04:43.9132809Z fi 2024-06-26T07:04:43.9139827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.9140377Z env: 2024-06-26T07:04:43.9140702Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.9141076Z ##[endgroup] 2024-06-26T07:04:43.9178024Z Docker daemon is running... 2024-06-26T07:04:43.9227436Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:04:43.9228086Z with: 2024-06-26T07:04:43.9228455Z shell: bash 2024-06-26T07:04:43.9228858Z timeout_minutes: 5 2024-06-26T07:04:43.9229278Z max_attempts: 3 2024-06-26T07:04:43.9229691Z retry_wait_seconds: 30 2024-06-26T07:04:43.9231428Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:04:43.9232951Z polling_interval_seconds: 1 2024-06-26T07:04:43.9233425Z warning_on_retry: true 2024-06-26T07:04:43.9233872Z continue_on_error: false 2024-06-26T07:04:43.9234301Z env: 2024-06-26T07:04:43.9234739Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.9235200Z AWS_RETRY_MODE: standard 2024-06-26T07:04:43.9235661Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:04:43.9236089Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:04:43.9236552Z ##[endgroup] 2024-06-26T07:04:44.7775554Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:44.7776443Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:44.7777287Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:44.7777822Z 2024-06-26T07:04:44.7777971Z Login Succeeded 2024-06-26T07:04:44.9741706Z Command completed after 1 attempt(s). 2024-06-26T07:04:44.9785243Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:44.9785974Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:44.9786645Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:44.9794489Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:44.9795113Z env: 2024-06-26T07:04:44.9795422Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:44.9795795Z ##[endgroup] 2024-06-26T07:04:44.9855468Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:44.9856258Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:44.9856884Z # shellcheck disable=SC2046 2024-06-26T07:04:44.9857366Z docker stop $(docker ps -q) || true 2024-06-26T07:04:44.9857883Z # Prune all of the docker images 2024-06-26T07:04:44.9858363Z docker system prune -af 2024-06-26T07:04:44.9866247Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:44.9866769Z env: 2024-06-26T07:04:44.9867072Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:44.9867446Z ##[endgroup] 2024-06-26T07:04:45.0110805Z "docker stop" requires at least 1 argument. 2024-06-26T07:04:45.0111522Z See 'docker stop --help'. 2024-06-26T07:04:45.0111775Z 2024-06-26T07:04:45.0112012Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:04:45.0112400Z 2024-06-26T07:04:45.0112562Z Stop one or more running containers 2024-06-26T07:04:45.0257209Z Total reclaimed space: 0B 2024-06-26T07:04:45.0285901Z ##[group]Run set +e 2024-06-26T07:04:45.0286279Z set +e 2024-06-26T07:04:45.0286729Z set -x 2024-06-26T07:04:45.0287052Z  2024-06-26T07:04:45.0287401Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:45.0288224Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:04:45.0289318Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:04:45.0290085Z # one is returned at random 2024-06-26T07:04:45.0290654Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:04:45.0291184Z  2024-06-26T07:04:45.0291531Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:45.0292171Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:04:45.0292954Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:04:45.0293527Z  2024-06-26T07:04:45.0293880Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:45.0294463Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:04:45.0294999Z  exit 1 2024-06-26T07:04:45.0295332Z  fi 2024-06-26T07:04:45.0295648Z fi 2024-06-26T07:04:45.0296048Z  2024-06-26T07:04:45.0296440Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:04:45.0296979Z  # Clean up any old records first 2024-06-26T07:04:45.0297511Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:04:45.0297993Z fi 2024-06-26T07:04:45.0298292Z  2024-06-26T07:04:45.0298749Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:04:45.0299314Z cat /etc/hosts 2024-06-26T07:04:45.0307144Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:45.0307669Z env: 2024-06-26T07:04:45.0307979Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:45.0308351Z ##[endgroup] 2024-06-26T07:04:45.0328910Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:45.0332746Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:04:45.0333207Z ++ tail -n1 2024-06-26T07:04:45.0427015Z + RESOLVED_IP=18.160.10.36 2024-06-26T07:04:45.0427460Z + '[' -z 18.160.10.36 ']' 2024-06-26T07:04:45.0427930Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:04:45.0433856Z 18.160.18.34 download.pytorch.org 2024-06-26T07:04:45.0434823Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:04:45.0530514Z + echo '18.160.10.36 download.pytorch.org' 2024-06-26T07:04:45.0531185Z + sudo tee -a /etc/hosts 2024-06-26T07:04:45.0834857Z 18.160.10.36 download.pytorch.org 2024-06-26T07:04:45.0848237Z + cat /etc/hosts 2024-06-26T07:04:45.0854316Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:04:45.0867162Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:04:45.0867691Z 18.160.10.36 download.pytorch.org 2024-06-26T07:04:45.1017490Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:04:45.1018202Z with: 2024-06-26T07:04:45.1019354Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1020649Z docker-build-dir: .ci/docker 2024-06-26T07:04:45.1021074Z working-directory: . 2024-06-26T07:04:45.1076608Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.1077550Z force-push: false 2024-06-26T07:04:45.1077932Z env: 2024-06-26T07:04:45.1078237Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:45.1078606Z ##[endgroup] 2024-06-26T07:04:45.1098376Z ##[group]Run set -ex 2024-06-26T07:04:45.1098761Z set -ex 2024-06-26T07:04:45.1099077Z  2024-06-26T07:04:45.1099641Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:04:45.1100609Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:04:45.1101606Z # job could then download the pre-built image as usual 2024-06-26T07:04:45.1102349Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:04:45.1103029Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1103673Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1104241Z  2024-06-26T07:04:45.1104758Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:04:45.1105392Z  exit 0 2024-06-26T07:04:45.1105707Z else 2024-06-26T07:04:45.1106092Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1106554Z fi 2024-06-26T07:04:45.1106847Z  2024-06-26T07:04:45.1107325Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:04:45.1108174Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:04:45.1108958Z  # use it as it is, but first let's extract the tag 2024-06-26T07:04:45.1109662Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:04:45.1110391Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1111104Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1111723Z else 2024-06-26T07:04:45.1112171Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:04:45.1112839Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1113736Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1114502Z fi 2024-06-26T07:04:45.1122825Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:45.1123336Z env: 2024-06-26T07:04:45.1123641Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:45.1124008Z REPO_NAME: pytorch 2024-06-26T07:04:45.1125132Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1126332Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:45.1126848Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.1127384Z ##[endgroup] 2024-06-26T07:04:45.1152378Z + [[ ! -d .ci/docker ]] 2024-06-26T07:04:45.1152824Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:04:45.1153234Z + echo skip=false 2024-06-26T07:04:45.1155339Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:04:45.1159256Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1161018Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:04:45.1164617Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1165765Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1167254Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1194385Z ##[group]Run set +e 2024-06-26T07:04:45.1194925Z set +e 2024-06-26T07:04:45.1195258Z set -x 2024-06-26T07:04:45.1195576Z  2024-06-26T07:04:45.1195880Z login() { 2024-06-26T07:04:45.1196567Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:45.1197506Z } 2024-06-26T07:04:45.1197843Z  2024-06-26T07:04:45.1198288Z retry () { 2024-06-26T07:04:45.1198708Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:45.1199185Z } 2024-06-26T07:04:45.1199484Z  2024-06-26T07:04:45.1199833Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:45.1200276Z  2024-06-26T07:04:45.1200770Z # Check if image already exists, if it does then skip building it 2024-06-26T07:04:45.1201494Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:04:45.1202022Z  exit 0 2024-06-26T07:04:45.1202364Z fi 2024-06-26T07:04:45.1202663Z  2024-06-26T07:04:45.1203196Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:04:45.1204032Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:04:45.1204795Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:04:45.1205474Z  # if we're on the base branch then use the parent commit 2024-06-26T07:04:45.1206094Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:04:45.1206542Z else 2024-06-26T07:04:45.1207034Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:04:45.1207734Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:04:45.1208263Z fi 2024-06-26T07:04:45.1208568Z  2024-06-26T07:04:45.1208921Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:04:45.1209438Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1209919Z  2024-06-26T07:04:45.1210590Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:04:45.1211367Z  exit 0 2024-06-26T07:04:45.1211689Z fi 2024-06-26T07:04:45.1211996Z  2024-06-26T07:04:45.1212460Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:04:45.1213439Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:04:45.1214249Z  exit 1 2024-06-26T07:04:45.1214581Z fi 2024-06-26T07:04:45.1214887Z  2024-06-26T07:04:45.1215395Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:04:45.1216347Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:04:45.1217231Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:04:45.1218202Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:04:45.1219306Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:04:45.1220059Z fi 2024-06-26T07:04:45.1220363Z  2024-06-26T07:04:45.1220757Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:45.1228519Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:45.1229040Z env: 2024-06-26T07:04:45.1229342Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:45.1229732Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:45.1230222Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:45.1231500Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1232748Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.1233363Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.1233907Z ##[endgroup] 2024-06-26T07:04:45.1253488Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.1254178Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.1255704Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:45.1256600Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.5400074Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:45.5400901Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:45.5401746Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:45.5402270Z 2024-06-26T07:04:45.5402408Z Login Succeeded 2024-06-26T07:04:45.5411207Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.7578395Z { 2024-06-26T07:04:45.7579213Z "schemaVersion": 2, 2024-06-26T07:04:45.7579991Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:04:45.7580715Z "config": { 2024-06-26T07:04:45.7581190Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:04:45.7581779Z "size": 48681, 2024-06-26T07:04:45.7582498Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:04:45.7583346Z }, 2024-06-26T07:04:45.7583755Z "layers": [ 2024-06-26T07:04:45.7584186Z { 2024-06-26T07:04:45.7584814Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7585546Z "size": 28580681, 2024-06-26T07:04:45.7586122Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:04:45.7587339Z }, 2024-06-26T07:04:45.7587793Z { 2024-06-26T07:04:45.7588461Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7589277Z "size": 7943451, 2024-06-26T07:04:45.7590060Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:04:45.7590984Z + exit 0 2024-06-26T07:04:45.7591344Z }, 2024-06-26T07:04:45.7591713Z { 2024-06-26T07:04:45.7592409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7593211Z "size": 55728572, 2024-06-26T07:04:45.7593851Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:04:45.7594479Z }, 2024-06-26T07:04:45.7594893Z { 2024-06-26T07:04:45.7595359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7595936Z "size": 186, 2024-06-26T07:04:45.7596498Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:04:45.7597138Z }, 2024-06-26T07:04:45.7597715Z { 2024-06-26T07:04:45.7598203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7598782Z "size": 6886, 2024-06-26T07:04:45.7599349Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:04:45.7600170Z }, 2024-06-26T07:04:45.7600446Z { 2024-06-26T07:04:45.7600901Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7601490Z "size": 1329001756, 2024-06-26T07:04:45.7602087Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:04:45.7602726Z }, 2024-06-26T07:04:45.7603004Z { 2024-06-26T07:04:45.7603455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7604015Z "size": 62501, 2024-06-26T07:04:45.7604577Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:04:45.7605207Z }, 2024-06-26T07:04:45.7605482Z { 2024-06-26T07:04:45.7605925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7606497Z "size": 1684, 2024-06-26T07:04:45.7607051Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:04:45.7607674Z }, 2024-06-26T07:04:45.7607954Z { 2024-06-26T07:04:45.7608407Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7609088Z "size": 1523, 2024-06-26T07:04:45.7609668Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:04:45.7610312Z }, 2024-06-26T07:04:45.7610588Z { 2024-06-26T07:04:45.7611088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7611656Z "size": 2528295403, 2024-06-26T07:04:45.7612239Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:04:45.7612875Z }, 2024-06-26T07:04:45.7613153Z { 2024-06-26T07:04:45.7613597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7614175Z "size": 86016, 2024-06-26T07:04:45.7614741Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:04:45.7615380Z }, 2024-06-26T07:04:45.7615650Z { 2024-06-26T07:04:45.7616109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7616687Z "size": 1823, 2024-06-26T07:04:45.7617230Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:04:45.7617857Z }, 2024-06-26T07:04:45.7618136Z { 2024-06-26T07:04:45.7618586Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7619152Z "size": 246737542, 2024-06-26T07:04:45.7619722Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:04:45.7620347Z }, 2024-06-26T07:04:45.7620623Z { 2024-06-26T07:04:45.7621072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7621642Z "size": 545, 2024-06-26T07:04:45.7622204Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:04:45.7622839Z }, 2024-06-26T07:04:45.7623107Z { 2024-06-26T07:04:45.7623560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7624129Z "size": 1283, 2024-06-26T07:04:45.7624676Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:04:45.7625293Z }, 2024-06-26T07:04:45.7625570Z { 2024-06-26T07:04:45.7626020Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7626582Z "size": 485, 2024-06-26T07:04:45.7627150Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:04:45.7627793Z }, 2024-06-26T07:04:45.7628071Z { 2024-06-26T07:04:45.7628519Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7629094Z "size": 91710937, 2024-06-26T07:04:45.7629684Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:04:45.7630328Z }, 2024-06-26T07:04:45.7630598Z { 2024-06-26T07:04:45.7631100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7631734Z "size": 3233, 2024-06-26T07:04:45.7632297Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:04:45.7632945Z }, 2024-06-26T07:04:45.7633224Z { 2024-06-26T07:04:45.7633673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7634235Z "size": 1905, 2024-06-26T07:04:45.7634938Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:04:45.7635593Z }, 2024-06-26T07:04:45.7635873Z { 2024-06-26T07:04:45.7636316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7636887Z "size": 702, 2024-06-26T07:04:45.7637642Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:45.7638285Z }, 2024-06-26T07:04:45.7638564Z { 2024-06-26T07:04:45.7639017Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7639587Z "size": 2784789002, 2024-06-26T07:04:45.7640167Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:04:45.7640899Z }, 2024-06-26T07:04:45.7641178Z { 2024-06-26T07:04:45.7641626Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7642184Z "size": 381, 2024-06-26T07:04:45.7642738Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:04:45.7643370Z }, 2024-06-26T07:04:45.7643640Z { 2024-06-26T07:04:45.7644094Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7644662Z "size": 11879, 2024-06-26T07:04:45.7645218Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:04:45.7645828Z }, 2024-06-26T07:04:45.7646105Z { 2024-06-26T07:04:45.7646557Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7647121Z "size": 803, 2024-06-26T07:04:45.7647681Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:04:45.7648324Z }, 2024-06-26T07:04:45.7648599Z { 2024-06-26T07:04:45.7649051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7649619Z "size": 106, 2024-06-26T07:04:45.7650168Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:04:45.7650840Z }, 2024-06-26T07:04:45.7651108Z { 2024-06-26T07:04:45.7651563Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7652136Z "size": 503, 2024-06-26T07:04:45.7652695Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:04:45.7653319Z }, 2024-06-26T07:04:45.7653596Z { 2024-06-26T07:04:45.7654050Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7654622Z "size": 121477177, 2024-06-26T07:04:45.7655195Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:04:45.7655821Z }, 2024-06-26T07:04:45.7656100Z { 2024-06-26T07:04:45.7656549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7657117Z "size": 109, 2024-06-26T07:04:45.7657679Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:04:45.7658315Z }, 2024-06-26T07:04:45.7658584Z { 2024-06-26T07:04:45.7659036Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7659603Z "size": 491, 2024-06-26T07:04:45.7660152Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:04:45.7660771Z }, 2024-06-26T07:04:45.7661050Z { 2024-06-26T07:04:45.7661502Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7662066Z "size": 291, 2024-06-26T07:04:45.7662623Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:04:45.7663376Z }, 2024-06-26T07:04:45.7663659Z { 2024-06-26T07:04:45.7664104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7664674Z "size": 101, 2024-06-26T07:04:45.7665225Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:04:45.7665849Z }, 2024-06-26T07:04:45.7666118Z { 2024-06-26T07:04:45.7666566Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7667135Z "size": 1473, 2024-06-26T07:04:45.7667685Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:04:45.7668299Z }, 2024-06-26T07:04:45.7668576Z { 2024-06-26T07:04:45.7669031Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7669598Z "size": 446710781, 2024-06-26T07:04:45.7670176Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:04:45.7670800Z }, 2024-06-26T07:04:45.7671083Z { 2024-06-26T07:04:45.7671527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7672165Z "size": 161, 2024-06-26T07:04:45.7672706Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:04:45.7673321Z }, 2024-06-26T07:04:45.7673591Z { 2024-06-26T07:04:45.7674044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7674727Z "size": 564, 2024-06-26T07:04:45.7675312Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:04:45.7675933Z }, 2024-06-26T07:04:45.7676209Z { 2024-06-26T07:04:45.7676662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7677224Z "size": 35874600, 2024-06-26T07:04:45.7678031Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:04:45.7678665Z }, 2024-06-26T07:04:45.7678943Z { 2024-06-26T07:04:45.7679397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7679972Z "size": 104, 2024-06-26T07:04:45.7680509Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:04:45.7681124Z }, 2024-06-26T07:04:45.7681393Z { 2024-06-26T07:04:45.7681843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7682409Z "size": 425, 2024-06-26T07:04:45.7682952Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:04:45.7683574Z }, 2024-06-26T07:04:45.7683851Z { 2024-06-26T07:04:45.7684303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7684866Z "size": 20262084, 2024-06-26T07:04:45.7685447Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:04:45.7686080Z }, 2024-06-26T07:04:45.7686359Z { 2024-06-26T07:04:45.7686807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7687376Z "size": 439, 2024-06-26T07:04:45.7687925Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:04:45.7688542Z }, 2024-06-26T07:04:45.7688817Z { 2024-06-26T07:04:45.7689268Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7689834Z "size": 702, 2024-06-26T07:04:45.7690389Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:45.7691024Z }, 2024-06-26T07:04:45.7691304Z { 2024-06-26T07:04:45.7691763Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7692326Z "size": 143, 2024-06-26T07:04:45.7692878Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:04:45.7693504Z }, 2024-06-26T07:04:45.7693779Z { 2024-06-26T07:04:45.7694223Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7694896Z "size": 135, 2024-06-26T07:04:45.7695467Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:04:45.7696116Z }, 2024-06-26T07:04:45.7696389Z { 2024-06-26T07:04:45.7696840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7697413Z "size": 4911495247, 2024-06-26T07:04:45.7697980Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:04:45.7698602Z }, 2024-06-26T07:04:45.7698881Z { 2024-06-26T07:04:45.7699333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7699892Z "size": 190, 2024-06-26T07:04:45.7700461Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:04:45.7701106Z }, 2024-06-26T07:04:45.7701383Z { 2024-06-26T07:04:45.7701829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7702400Z "size": 565, 2024-06-26T07:04:45.7702957Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:04:45.7703658Z }, 2024-06-26T07:04:45.7703935Z { 2024-06-26T07:04:45.7704386Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7704961Z "size": 43163095, 2024-06-26T07:04:45.7705533Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:04:45.7706164Z }, 2024-06-26T07:04:45.7706442Z { 2024-06-26T07:04:45.7706900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7707463Z "size": 106, 2024-06-26T07:04:45.7708017Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:04:45.7708647Z }, 2024-06-26T07:04:45.7708917Z { 2024-06-26T07:04:45.7709374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7709972Z "size": 1213, 2024-06-26T07:04:45.7710534Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:04:45.7711151Z }, 2024-06-26T07:04:45.7711433Z { 2024-06-26T07:04:45.7711882Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7712449Z "size": 702, 2024-06-26T07:04:45.7713000Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:45.7713633Z }, 2024-06-26T07:04:45.7713912Z { 2024-06-26T07:04:45.7714365Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7715015Z "size": 138, 2024-06-26T07:04:45.7715581Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:04:45.7716224Z }, 2024-06-26T07:04:45.7716492Z { 2024-06-26T07:04:45.7716943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7717682Z "size": 120, 2024-06-26T07:04:45.7718257Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:04:45.7718885Z }, 2024-06-26T07:04:45.7719161Z { 2024-06-26T07:04:45.7719622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7720197Z "size": 1904157495, 2024-06-26T07:04:45.7720787Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:04:45.7721474Z }, 2024-06-26T07:04:45.7721754Z { 2024-06-26T07:04:45.7722207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7722769Z "size": 172, 2024-06-26T07:04:45.7723327Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:04:45.7723966Z }, 2024-06-26T07:04:45.7724240Z { 2024-06-26T07:04:45.7724690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7725257Z "size": 908, 2024-06-26T07:04:45.7725825Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:04:45.7726579Z }, 2024-06-26T07:04:45.7726865Z { 2024-06-26T07:04:45.7727318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7727890Z "size": 702, 2024-06-26T07:04:45.7728442Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:45.7729075Z }, 2024-06-26T07:04:45.7729352Z { 2024-06-26T07:04:45.7729792Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7730359Z "size": 134, 2024-06-26T07:04:45.7730916Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:04:45.7731582Z }, 2024-06-26T07:04:45.7731852Z { 2024-06-26T07:04:45.7732304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7732872Z "size": 32, 2024-06-26T07:04:45.7733431Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:45.7734053Z }, 2024-06-26T07:04:45.7734337Z { 2024-06-26T07:04:45.7734791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7735447Z "size": 156, 2024-06-26T07:04:45.7736008Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:04:45.7736648Z }, 2024-06-26T07:04:45.7736925Z { 2024-06-26T07:04:45.7737369Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7737938Z "size": 1841, 2024-06-26T07:04:45.7738506Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:04:45.7739143Z }, 2024-06-26T07:04:45.7739411Z { 2024-06-26T07:04:45.7739868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7740437Z "size": 7529784, 2024-06-26T07:04:45.7741008Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:04:45.7741630Z }, 2024-06-26T07:04:45.7741905Z { 2024-06-26T07:04:45.7742364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7742923Z "size": 165, 2024-06-26T07:04:45.7743482Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:04:45.7744111Z }, 2024-06-26T07:04:45.7744388Z { 2024-06-26T07:04:45.7744831Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7745398Z "size": 7944, 2024-06-26T07:04:45.7745962Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:04:45.7746596Z }, 2024-06-26T07:04:45.7746866Z { 2024-06-26T07:04:45.7747315Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7747886Z "size": 8070, 2024-06-26T07:04:45.7748440Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:04:45.7749057Z }, 2024-06-26T07:04:45.7749333Z { 2024-06-26T07:04:45.7749791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7750355Z "size": 303, 2024-06-26T07:04:45.7750913Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:04:45.7751552Z }, 2024-06-26T07:04:45.7751829Z { 2024-06-26T07:04:45.7752275Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7752850Z "size": 7629071, 2024-06-26T07:04:45.7753397Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:04:45.7754004Z }, 2024-06-26T07:04:45.7754277Z { 2024-06-26T07:04:45.7754811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7755381Z "size": 108, 2024-06-26T07:04:45.7755934Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:04:45.7756568Z }, 2024-06-26T07:04:45.7756843Z { 2024-06-26T07:04:45.7757468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7758248Z "size": 54145775, 2024-06-26T07:04:45.7758883Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:04:45.7759582Z }, 2024-06-26T07:04:45.7759848Z { 2024-06-26T07:04:45.7760316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7760920Z "size": 474, 2024-06-26T07:04:45.7761516Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:04:45.7762203Z }, 2024-06-26T07:04:45.7762463Z { 2024-06-26T07:04:45.7762933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7763546Z "size": 1374858747, 2024-06-26T07:04:45.7764149Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:04:45.7764822Z }, 2024-06-26T07:04:45.7765090Z { 2024-06-26T07:04:45.7765561Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7766159Z "size": 106, 2024-06-26T07:04:45.7766769Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:04:45.7767534Z }, 2024-06-26T07:04:45.7767813Z { 2024-06-26T07:04:45.7768256Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7768825Z "size": 558, 2024-06-26T07:04:45.7769388Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:04:45.7770031Z }, 2024-06-26T07:04:45.7770301Z { 2024-06-26T07:04:45.7770752Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7771324Z "size": 46248513, 2024-06-26T07:04:45.7771882Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:04:45.7772502Z }, 2024-06-26T07:04:45.7772780Z { 2024-06-26T07:04:45.7773231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7773792Z "size": 111, 2024-06-26T07:04:45.7774348Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:04:45.7774974Z }, 2024-06-26T07:04:45.7775257Z { 2024-06-26T07:04:45.7775703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7776271Z "size": 32, 2024-06-26T07:04:45.7776829Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:45.7777457Z }, 2024-06-26T07:04:45.7777737Z { 2024-06-26T07:04:45.7778187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7778756Z "size": 32, 2024-06-26T07:04:45.7779306Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:45.7779945Z }, 2024-06-26T07:04:45.7780222Z { 2024-06-26T07:04:45.7780672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7781231Z "size": 32, 2024-06-26T07:04:45.7781788Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:45.7782422Z }, 2024-06-26T07:04:45.7782691Z { 2024-06-26T07:04:45.7783148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:45.7783714Z "size": 32, 2024-06-26T07:04:45.7784269Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:45.7784893Z } 2024-06-26T07:04:45.7785170Z ] 2024-06-26T07:04:45.7785443Z } 2024-06-26T07:04:45.7888006Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:45.7888530Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:45.7889093Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:04:45.7897076Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:45.7897601Z env: 2024-06-26T07:04:45.7897910Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:45.7899069Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.7900248Z ##[endgroup] 2024-06-26T07:04:45.7936870Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.7985300Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:04:45.7985900Z with: 2024-06-26T07:04:45.7986984Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.7988308Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.7988836Z env: 2024-06-26T07:04:45.7989141Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:45.7989514Z ##[endgroup] 2024-06-26T07:04:45.8006027Z ##[group]Run set -x 2024-06-26T07:04:45.8006392Z set -x 2024-06-26T07:04:45.8006719Z set +e 2024-06-26T07:04:45.8007037Z  2024-06-26T07:04:45.8007338Z login() { 2024-06-26T07:04:45.8008011Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:45.8008847Z } 2024-06-26T07:04:45.8009152Z  2024-06-26T07:04:45.8009481Z retry () { 2024-06-26T07:04:45.8009902Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:45.8010372Z } 2024-06-26T07:04:45.8010678Z  2024-06-26T07:04:45.8011024Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:45.8011457Z  2024-06-26T07:04:45.8011758Z set -e 2024-06-26T07:04:45.8012255Z # ignore output since only exit code is used for conditional 2024-06-26T07:04:45.8012970Z # only pull docker image if it's not available locally 2024-06-26T07:04:45.8013739Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:04:45.8014448Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:04:45.8014913Z fi 2024-06-26T07:04:45.8021913Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:45.8022430Z env: 2024-06-26T07:04:45.8022737Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:45.8023890Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.8025192Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.8025727Z ##[endgroup] 2024-06-26T07:04:45.8044510Z + set +e 2024-06-26T07:04:45.8045199Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.8045852Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.8046815Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:45.8047756Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:46.2230776Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:46.2231693Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:46.2232723Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:46.2233265Z 2024-06-26T07:04:46.2233405Z Login Succeeded 2024-06-26T07:04:46.2241689Z + set -e 2024-06-26T07:04:46.2243025Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:46.2362195Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:46.2365234Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:46.4234946Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:04:46.4236390Z 7a2c55901189: Pulling fs layer 2024-06-26T07:04:46.4236952Z 224fe954d725: Pulling fs layer 2024-06-26T07:04:46.4237972Z 75722010b82e: Pulling fs layer 2024-06-26T07:04:46.4238567Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:04:46.4239136Z b57676e46aee: Pulling fs layer 2024-06-26T07:04:46.4239628Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:04:46.4240149Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:04:46.4240598Z 0c1227890755: Pulling fs layer 2024-06-26T07:04:46.4240994Z d8d1234baab3: Pulling fs layer 2024-06-26T07:04:46.4241389Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:04:46.4241792Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:04:46.4242190Z 309b872da805: Pulling fs layer 2024-06-26T07:04:46.4242578Z e82932cb002c: Pulling fs layer 2024-06-26T07:04:46.4242975Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:04:46.4243370Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:04:46.4243763Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:04:46.4244221Z f4863801e30c: Pulling fs layer 2024-06-26T07:04:46.4244736Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:04:46.4245133Z cf2f34f225db: Pulling fs layer 2024-06-26T07:04:46.4245545Z df546d7b3e34: Pulling fs layer 2024-06-26T07:04:46.4245936Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:04:46.4246595Z 96d0e2032184: Pulling fs layer 2024-06-26T07:04:46.4246988Z 06e12ce59790: Pulling fs layer 2024-06-26T07:04:46.4247383Z ed712f63cde7: Pulling fs layer 2024-06-26T07:04:46.4247775Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:04:46.4248174Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:04:46.4248568Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:04:46.4248967Z de45a14c9b22: Pulling fs layer 2024-06-26T07:04:46.4249353Z e99241e77209: Pulling fs layer 2024-06-26T07:04:46.4249749Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:04:46.4250246Z a823ffa4c541: Pulling fs layer 2024-06-26T07:04:46.4250626Z 309b872da805: Waiting 2024-06-26T07:04:46.4250991Z 44ba272eae20: Pulling fs layer 2024-06-26T07:04:46.4251386Z da02d278e524: Pulling fs layer 2024-06-26T07:04:46.4251791Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:04:46.4252167Z e82932cb002c: Waiting 2024-06-26T07:04:46.4252517Z df546d7b3e34: Waiting 2024-06-26T07:04:46.4252879Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:04:46.4253259Z 3d914950f8ad: Waiting 2024-06-26T07:04:46.4253610Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:04:46.4254005Z b2950a036534: Pulling fs layer 2024-06-26T07:04:46.4254385Z ed712f63cde7: Waiting 2024-06-26T07:04:46.4254736Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:04:46.4255136Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:04:46.4255518Z a97a71ef8a4e: Waiting 2024-06-26T07:04:46.4255875Z 7a868239227f: Pulling fs layer 2024-06-26T07:04:46.4256248Z 96d0e2032184: Waiting 2024-06-26T07:04:46.4256595Z ca9dd73e3e93: Waiting 2024-06-26T07:04:46.4257121Z 06e12ce59790: Waiting 2024-06-26T07:04:46.4257479Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:04:46.4257878Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:04:46.4258279Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:04:46.4258678Z c1068d03a811: Pulling fs layer 2024-06-26T07:04:46.4259074Z 671b4a301d81: Pulling fs layer 2024-06-26T07:04:46.4259477Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:04:46.4259867Z a8c1e85b5e14: Waiting 2024-06-26T07:04:46.4260287Z 35e8f4e2a42f: Waiting 2024-06-26T07:04:46.4260683Z a41a8d1c11c8: Waiting 2024-06-26T07:04:46.4261019Z 9bf94bc45658: Waiting 2024-06-26T07:04:46.4261358Z 4bdf41c9ecbe: Waiting 2024-06-26T07:04:46.4261751Z 0c1227890755: Waiting 2024-06-26T07:04:46.4262090Z a823ffa4c541: Waiting 2024-06-26T07:04:46.4262451Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:04:46.4262837Z de45a14c9b22: Waiting 2024-06-26T07:04:46.4263247Z fb2e97768560: Pulling fs layer 2024-06-26T07:04:46.4263909Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:04:46.4264295Z cf2f34f225db: Waiting 2024-06-26T07:04:46.4264654Z e08e88c13668: Pulling fs layer 2024-06-26T07:04:46.4265043Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:04:46.4265428Z d527cbbb87e3: Waiting 2024-06-26T07:04:46.4265768Z 16d76a79d5d6: Waiting 2024-06-26T07:04:46.4266123Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:04:46.4266509Z b57676e46aee: Waiting 2024-06-26T07:04:46.4266950Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:04:46.4267341Z b2950a036534: Waiting 2024-06-26T07:04:46.4267671Z 73a0ce681c29: Waiting 2024-06-26T07:04:46.4268025Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:04:46.4268422Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:04:46.4268804Z f4863801e30c: Waiting 2024-06-26T07:04:46.4269151Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:04:46.4269551Z e797fe86caef: Pulling fs layer 2024-06-26T07:04:46.4269932Z 83911d7c7ecf: Waiting 2024-06-26T07:04:46.4270279Z 967190076fc0: Pulling fs layer 2024-06-26T07:04:46.4270799Z 9cd4c745f8e4: Waiting 2024-06-26T07:04:46.4271201Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:04:46.4271586Z 25e19d81d9fc: Waiting 2024-06-26T07:04:46.4271936Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:04:46.4272320Z 7a868239227f: Waiting 2024-06-26T07:04:46.4272675Z cc91933e985a: Pulling fs layer 2024-06-26T07:04:46.4273063Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:04:46.4273526Z 7ed32bc8e469: Waiting 2024-06-26T07:04:46.4273875Z 526083994857: Pulling fs layer 2024-06-26T07:04:46.4274272Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:04:46.4274769Z b3c5d66c679f: Waiting 2024-06-26T07:04:46.4275130Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:04:46.4275516Z d8d1234baab3: Waiting 2024-06-26T07:04:46.4275874Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:04:46.4276249Z c1068d03a811: Waiting 2024-06-26T07:04:46.4276590Z e99241e77209: Waiting 2024-06-26T07:04:46.4276939Z 95237e7311a1: Pulling fs layer 2024-06-26T07:04:46.4277440Z e08e88c13668: Waiting 2024-06-26T07:04:46.4277895Z 67760cf5ec46: Waiting 2024-06-26T07:04:46.4278300Z 671b4a301d81: Waiting 2024-06-26T07:04:46.4278655Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:04:46.4279031Z 44ba272eae20: Waiting 2024-06-26T07:04:46.4279368Z da02d278e524: Waiting 2024-06-26T07:04:46.4279710Z 12ba7431ff38: Waiting 2024-06-26T07:04:46.4280065Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:04:46.4280447Z 49737bd5ae70: Waiting 2024-06-26T07:04:46.4280787Z 05a6f6dcc1c0: Waiting 2024-06-26T07:04:46.4281125Z d37dd9ed8948: Waiting 2024-06-26T07:04:46.4281468Z e797fe86caef: Waiting 2024-06-26T07:04:46.4281807Z ddc9740bcea2: Waiting 2024-06-26T07:04:46.4282150Z ec1e7978c1fe: Waiting 2024-06-26T07:04:46.4282491Z 68af2ec4e30a: Waiting 2024-06-26T07:04:46.4282825Z cf7c2cb896cb: Waiting 2024-06-26T07:04:46.4283165Z 9f3187916e7d: Waiting 2024-06-26T07:04:46.4283501Z cc91933e985a: Waiting 2024-06-26T07:04:46.4283834Z 3a778ddfac2a: Waiting 2024-06-26T07:04:46.4284167Z 967190076fc0: Waiting 2024-06-26T07:04:46.4284498Z 526083994857: Waiting 2024-06-26T07:04:46.4284829Z 6bc9dc8315b0: Waiting 2024-06-26T07:04:46.4285174Z 2060028f7488: Pulling fs layer 2024-06-26T07:04:46.4285552Z fb2e97768560: Waiting 2024-06-26T07:04:46.4285904Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:04:46.4286287Z 789eff6c51aa: Waiting 2024-06-26T07:04:46.4286618Z b599c5a8c7cb: Waiting 2024-06-26T07:04:46.4286957Z 84b2c7a89982: Waiting 2024-06-26T07:04:46.4287297Z 2060028f7488: Waiting 2024-06-26T07:04:46.4287623Z de0f8a8c45c6: Waiting 2024-06-26T07:04:46.4287962Z 95237e7311a1: Waiting 2024-06-26T07:04:46.4288294Z e412c7f3b2de: Waiting 2024-06-26T07:04:46.4288629Z 236c8f0245f4: Waiting 2024-06-26T07:04:46.4288958Z 4f4fb700ef54: Waiting 2024-06-26T07:04:46.4289292Z 25cd0c482ad6: Waiting 2024-06-26T07:04:46.5653362Z 224fe954d725: Verifying Checksum 2024-06-26T07:04:46.5653945Z 224fe954d725: Download complete 2024-06-26T07:04:46.6880155Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:04:46.6880814Z d527cbbb87e3: Download complete 2024-06-26T07:04:46.7560013Z 7a2c55901189: Verifying Checksum 2024-06-26T07:04:46.7560698Z 7a2c55901189: Download complete 2024-06-26T07:04:46.7618954Z b57676e46aee: Download complete 2024-06-26T07:04:46.8231931Z a41a8d1c11c8: Download complete 2024-06-26T07:04:46.8964470Z 0c1227890755: Download complete 2024-06-26T07:04:46.9709604Z d8d1234baab3: Verifying Checksum 2024-06-26T07:04:46.9710247Z d8d1234baab3: Download complete 2024-06-26T07:04:47.0409446Z 75722010b82e: Verifying Checksum 2024-06-26T07:04:47.0409925Z 75722010b82e: Download complete 2024-06-26T07:04:47.1247505Z ec1e7978c1fe: Download complete 2024-06-26T07:04:47.1899637Z 309b872da805: Download complete 2024-06-26T07:04:47.7908616Z 7a2c55901189: Pull complete 2024-06-26T07:04:48.0818608Z 224fe954d725: Pull complete 2024-06-26T07:04:49.0115354Z 75722010b82e: Pull complete 2024-06-26T07:04:49.0612612Z d527cbbb87e3: Pull complete 2024-06-26T07:04:49.1125129Z b57676e46aee: Pull complete 2024-06-26T07:04:49.7119607Z e82932cb002c: Verifying Checksum 2024-06-26T07:04:49.7120143Z e82932cb002c: Download complete 2024-06-26T07:04:49.8357846Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:04:49.8358449Z ca9dd73e3e93: Download complete 2024-06-26T07:04:49.9101084Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:04:49.9101670Z 16d76a79d5d6: Download complete 2024-06-26T07:04:50.0089459Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:04:50.0089954Z 4bdf41c9ecbe: Download complete 2024-06-26T07:04:50.9708734Z f4863801e30c: Verifying Checksum 2024-06-26T07:04:50.9709192Z f4863801e30c: Download complete 2024-06-26T07:04:51.0358069Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:04:51.0358508Z d37dd9ed8948: Download complete 2024-06-26T07:04:51.1063289Z cf2f34f225db: Verifying Checksum 2024-06-26T07:04:51.1063729Z cf2f34f225db: Download complete 2024-06-26T07:04:51.1734433Z df546d7b3e34: Verifying Checksum 2024-06-26T07:04:51.1734864Z df546d7b3e34: Download complete 2024-06-26T07:05:00.0964713Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:05:00.0965290Z a8c1e85b5e14: Download complete 2024-06-26T07:05:00.1604694Z 96d0e2032184: Verifying Checksum 2024-06-26T07:05:00.1605240Z 96d0e2032184: Download complete 2024-06-26T07:05:00.2296862Z 06e12ce59790: Download complete 2024-06-26T07:05:00.2954874Z ed712f63cde7: Download complete 2024-06-26T07:05:00.3586479Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:05:00.3587017Z a97a71ef8a4e: Download complete 2024-06-26T07:05:00.4281721Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:05:00.4282288Z 25cd0c482ad6: Download complete 2024-06-26T07:05:01.6932374Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:05:01.6932964Z 9bf94bc45658: Download complete 2024-06-26T07:05:01.7670808Z de45a14c9b22: Download complete 2024-06-26T07:05:01.8549597Z e99241e77209: Verifying Checksum 2024-06-26T07:05:01.8550192Z e99241e77209: Download complete 2024-06-26T07:05:01.9640408Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:05:01.9640898Z 35e8f4e2a42f: Download complete 2024-06-26T07:05:02.0649659Z a823ffa4c541: Verifying Checksum 2024-06-26T07:05:02.0650163Z a823ffa4c541: Download complete 2024-06-26T07:05:02.1344815Z 44ba272eae20: Verifying Checksum 2024-06-26T07:05:02.1345333Z 44ba272eae20: Download complete 2024-06-26T07:05:06.6557037Z da02d278e524: Verifying Checksum 2024-06-26T07:05:06.6557687Z da02d278e524: Download complete 2024-06-26T07:05:06.7254798Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:05:06.7255241Z 73a0ce681c29: Download complete 2024-06-26T07:05:06.7938094Z 49737bd5ae70: Download complete 2024-06-26T07:05:07.2047036Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:05:07.2047711Z 9cd4c745f8e4: Download complete 2024-06-26T07:05:07.3197134Z b2950a036534: Verifying Checksum 2024-06-26T07:05:07.3197917Z b2950a036534: Download complete 2024-06-26T07:05:07.4286627Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:05:07.4287165Z 25e19d81d9fc: Download complete 2024-06-26T07:05:07.6766640Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:05:07.6767117Z 60156b40fcbe: Download complete 2024-06-26T07:05:07.7361985Z 7a868239227f: Verifying Checksum 2024-06-26T07:05:07.7362558Z 7a868239227f: Download complete 2024-06-26T07:05:07.8071330Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:05:07.8071819Z 83911d7c7ecf: Download complete 2024-06-26T07:05:07.8787728Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:05:07.8788191Z b3c5d66c679f: Download complete 2024-06-26T07:05:12.3332028Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:05:12.3332513Z 7ed32bc8e469: Download complete 2024-06-26T07:05:12.4067089Z c1068d03a811: Download complete 2024-06-26T07:05:12.4744515Z 671b4a301d81: Download complete 2024-06-26T07:05:12.6984309Z a8c1e85b5e14: Pull complete 2024-06-26T07:05:12.9549552Z a41a8d1c11c8: Pull complete 2024-06-26T07:05:12.9618509Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:05:12.9618981Z 67760cf5ec46: Download complete 2024-06-26T07:05:13.0300284Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:05:13.0300763Z ddc9740bcea2: Download complete 2024-06-26T07:05:13.0905266Z fb2e97768560: Verifying Checksum 2024-06-26T07:05:13.0905775Z fb2e97768560: Download complete 2024-06-26T07:05:13.1395942Z 0c1227890755: Pull complete 2024-06-26T07:05:13.1626606Z b599c5a8c7cb: Download complete 2024-06-26T07:05:13.2470794Z e08e88c13668: Verifying Checksum 2024-06-26T07:05:13.2471242Z e08e88c13668: Download complete 2024-06-26T07:05:13.3741350Z d8d1234baab3: Pull complete 2024-06-26T07:05:19.0611273Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:05:19.0611909Z 3d914950f8ad: Download complete 2024-06-26T07:05:19.1336722Z cf7c2cb896cb: Download complete 2024-06-26T07:05:19.2073407Z 3a778ddfac2a: Download complete 2024-06-26T07:05:19.2815544Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:05:19.2816084Z 236c8f0245f4: Download complete 2024-06-26T07:05:19.2883843Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:05:19.2884290Z 4f4fb700ef54: Download complete 2024-06-26T07:05:19.3857613Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:05:19.3858131Z dda1ce302b9a: Download complete 2024-06-26T07:05:19.4815077Z e797fe86caef: Download complete 2024-06-26T07:05:19.6126208Z 967190076fc0: Verifying Checksum 2024-06-26T07:05:19.6126725Z 967190076fc0: Download complete 2024-06-26T07:05:19.6877624Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:05:19.6878116Z 4fb0a83f0685: Download complete 2024-06-26T07:05:19.7566839Z 68af2ec4e30a: Download complete 2024-06-26T07:05:19.8283832Z cc91933e985a: Download complete 2024-06-26T07:05:19.8982201Z 9f3187916e7d: Download complete 2024-06-26T07:05:20.0281748Z 526083994857: Verifying Checksum 2024-06-26T07:05:20.0282344Z 526083994857: Download complete 2024-06-26T07:05:20.0959771Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:05:20.0960326Z 05a6f6dcc1c0: Download complete 2024-06-26T07:05:20.6921621Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:05:20.6922163Z 6bc9dc8315b0: Download complete 2024-06-26T07:05:20.7642722Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:05:20.7643384Z 1b0e27c35f07: Download complete 2024-06-26T07:05:43.5676194Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:05:43.5676707Z 12ba7431ff38: Download complete 2024-06-26T07:05:43.6349513Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:05:43.6350050Z 789eff6c51aa: Download complete 2024-06-26T07:05:43.7162251Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:05:43.7162741Z de0f8a8c45c6: Download complete 2024-06-26T07:05:45.2886145Z 2060028f7488: Verifying Checksum 2024-06-26T07:05:45.2886648Z 2060028f7488: Download complete 2024-06-26T07:05:45.3692077Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:05:45.3692629Z e412c7f3b2de: Download complete 2024-06-26T07:05:48.8041011Z 95237e7311a1: Verifying Checksum 2024-06-26T07:05:48.8041522Z 95237e7311a1: Download complete 2024-06-26T07:06:11.4969946Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:06:11.4970441Z 84b2c7a89982: Download complete 2024-06-26T07:06:29.1892749Z 7ed32bc8e469: Pull complete 2024-06-26T07:06:29.4294609Z ec1e7978c1fe: Pull complete 2024-06-26T07:06:29.6600908Z 309b872da805: Pull complete 2024-06-26T07:06:37.0763095Z e82932cb002c: Pull complete 2024-06-26T07:06:37.2986629Z ca9dd73e3e93: Pull complete 2024-06-26T07:06:37.5231436Z 16d76a79d5d6: Pull complete 2024-06-26T07:06:37.7655786Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:06:40.1641672Z f4863801e30c: Pull complete 2024-06-26T07:06:40.4025368Z d37dd9ed8948: Pull complete 2024-06-26T07:06:40.6260690Z cf2f34f225db: Pull complete 2024-06-26T07:06:40.8370566Z df546d7b3e34: Pull complete 2024-06-26T07:07:30.2661825Z 3d914950f8ad: Pull complete 2024-06-26T07:07:30.4408570Z 96d0e2032184: Pull complete 2024-06-26T07:07:30.5385088Z 06e12ce59790: Pull complete 2024-06-26T07:07:30.6595639Z ed712f63cde7: Pull complete 2024-06-26T07:07:30.7688443Z a97a71ef8a4e: Pull complete 2024-06-26T07:07:30.8733020Z 25cd0c482ad6: Pull complete 2024-06-26T07:07:34.4174739Z 9bf94bc45658: Pull complete 2024-06-26T07:07:34.6418203Z de45a14c9b22: Pull complete 2024-06-26T07:07:34.7988527Z e99241e77209: Pull complete 2024-06-26T07:07:34.9241130Z 35e8f4e2a42f: Pull complete 2024-06-26T07:07:35.1111147Z a823ffa4c541: Pull complete 2024-06-26T07:07:35.3512342Z 44ba272eae20: Pull complete 2024-06-26T07:07:43.5852446Z da02d278e524: Pull complete 2024-06-26T07:07:43.7477931Z 73a0ce681c29: Pull complete 2024-06-26T07:07:43.9803716Z 49737bd5ae70: Pull complete 2024-06-26T07:07:44.9258672Z 9cd4c745f8e4: Pull complete 2024-06-26T07:07:45.1452060Z b2950a036534: Pull complete 2024-06-26T07:07:45.2986712Z 25e19d81d9fc: Pull complete 2024-06-26T07:07:45.7685534Z 60156b40fcbe: Pull complete 2024-06-26T07:07:45.9170355Z 7a868239227f: Pull complete 2024-06-26T07:07:46.2833874Z 83911d7c7ecf: Pull complete 2024-06-26T07:07:46.5073412Z b3c5d66c679f: Pull complete 2024-06-26T07:08:25.8637477Z 84b2c7a89982: Pull complete 2024-06-26T07:08:26.1020521Z c1068d03a811: Pull complete 2024-06-26T07:08:26.3266236Z 671b4a301d81: Pull complete 2024-06-26T07:08:28.3508351Z 67760cf5ec46: Pull complete 2024-06-26T07:08:28.3942882Z ddc9740bcea2: Pull complete 2024-06-26T07:08:28.4385510Z fb2e97768560: Pull complete 2024-06-26T07:08:28.5314359Z b599c5a8c7cb: Pull complete 2024-06-26T07:08:28.5747295Z e08e88c13668: Pull complete 2024-06-26T07:09:02.1963297Z 12ba7431ff38: Pull complete 2024-06-26T07:09:02.3975836Z cf7c2cb896cb: Pull complete 2024-06-26T07:09:02.5400607Z 3a778ddfac2a: Pull complete 2024-06-26T07:09:03.0166047Z 236c8f0245f4: Pull complete 2024-06-26T07:09:03.2531821Z 4f4fb700ef54: Pull complete 2024-06-26T07:09:03.4874881Z dda1ce302b9a: Pull complete 2024-06-26T07:09:03.7262198Z e797fe86caef: Pull complete 2024-06-26T07:09:04.0944534Z 967190076fc0: Pull complete 2024-06-26T07:09:04.3351352Z 4fb0a83f0685: Pull complete 2024-06-26T07:09:04.5558695Z 68af2ec4e30a: Pull complete 2024-06-26T07:09:04.7844836Z cc91933e985a: Pull complete 2024-06-26T07:09:05.0076304Z 9f3187916e7d: Pull complete 2024-06-26T07:09:06.1326555Z 526083994857: Pull complete 2024-06-26T07:09:06.3666603Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:09:08.4598897Z 6bc9dc8315b0: Pull complete 2024-06-26T07:09:08.6800453Z 1b0e27c35f07: Pull complete 2024-06-26T07:09:22.3112002Z 95237e7311a1: Pull complete 2024-06-26T07:09:22.3694924Z 789eff6c51aa: Pull complete 2024-06-26T07:09:22.4091539Z de0f8a8c45c6: Pull complete 2024-06-26T07:09:23.1112272Z 2060028f7488: Pull complete 2024-06-26T07:09:23.1654292Z e412c7f3b2de: Pull complete 2024-06-26T07:09:23.4432041Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:09:23.4465659Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:23.4500751Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:23.4554263Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:23.4555329Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:23.4563790Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:23.4564313Z env: 2024-06-26T07:09:23.4564625Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:23.4564994Z ##[endgroup] 2024-06-26T07:09:23.4720607Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:09:23.4721279Z with: 2024-06-26T07:09:23.4721596Z driver-version: 550.54.15 2024-06-26T07:09:23.4721972Z env: 2024-06-26T07:09:23.4722266Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:23.4722646Z ##[endgroup] 2024-06-26T07:09:23.4763800Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:09:23.4764395Z with: 2024-06-26T07:09:23.4764706Z timeout_minutes: 10 2024-06-26T07:09:23.4765053Z max_attempts: 3 2024-06-26T07:09:23.4796021Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:09:23.4827211Z retry_wait_seconds: 10 2024-06-26T07:09:23.4827606Z polling_interval_seconds: 1 2024-06-26T07:09:23.4827999Z warning_on_retry: true 2024-06-26T07:09:23.4828376Z continue_on_error: false 2024-06-26T07:09:23.4828734Z env: 2024-06-26T07:09:23.4829035Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:23.4829416Z DRIVER_VERSION: 550.54.15 2024-06-26T07:09:23.4829789Z ##[endgroup] 2024-06-26T07:09:23.5363103Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:09:23.5364070Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:09:23.5364776Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:09:23.8380852Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:23.8765941Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:09:23.9004169Z No Packages marked for removal 2024-06-26T07:09:23.9119257Z + install_nvidia_driver_common 2024-06-26T07:09:23.9121539Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:09:23.9122184Z + lspci 2024-06-26T07:09:23.9122521Z Before installing NVIDIA driver 2024-06-26T07:09:23.9199308Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:23.9200241Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:23.9201458Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:23.9202491Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:23.9203292Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:23.9204086Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:23.9204842Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:23.9205650Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:23.9206253Z + lsmod 2024-06-26T07:09:23.9212758Z Module Size Used by 2024-06-26T07:09:23.9213207Z xt_nat 16384 0 2024-06-26T07:09:23.9213669Z ib_core 266240 0 2024-06-26T07:09:23.9214145Z nvidia_modeset 1339392 0 2024-06-26T07:09:23.9214549Z veth 16384 0 2024-06-26T07:09:23.9214968Z nvidia_uvm 4599808 0 2024-06-26T07:09:23.9215605Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:23.9216207Z drm 421888 1 nvidia 2024-06-26T07:09:23.9216680Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:23.9217178Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:23.9217650Z xt_conntrack 16384 1 2024-06-26T07:09:23.9218087Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:23.9218577Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:23.9219074Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:23.9219618Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:23.9220137Z xfrm_user 45056 1 2024-06-26T07:09:23.9220638Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:23.9221092Z iptable_nat 16384 1 2024-06-26T07:09:23.9221490Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:23.9221951Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:23.9222456Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:23.9223078Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:23.9224427Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:23.9225492Z xt_addrtype 16384 2 2024-06-26T07:09:23.9225903Z iptable_filter 16384 1 2024-06-26T07:09:23.9226323Z br_netfilter 24576 0 2024-06-26T07:09:23.9226749Z bridge 172032 1 br_netfilter 2024-06-26T07:09:23.9227218Z stp 16384 1 bridge 2024-06-26T07:09:23.9227721Z llc 16384 2 bridge,stp 2024-06-26T07:09:23.9228171Z overlay 86016 0 2024-06-26T07:09:23.9228563Z sunrpc 393216 1 2024-06-26T07:09:23.9228959Z dm_mirror 28672 0 2024-06-26T07:09:23.9229380Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:23.9229888Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:23.9230424Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:23.9230887Z dax 69632 1 dm_mod 2024-06-26T07:09:23.9231315Z crc32_pclmul 16384 0 2024-06-26T07:09:23.9231832Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:23.9232231Z pcbc 16384 0 2024-06-26T07:09:23.9232682Z aesni_intel 188416 0 2024-06-26T07:09:23.9233104Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:23.9233583Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:23.9234058Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:23.9234633Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:23.9235344Z mousedev 24576 0 2024-06-26T07:09:23.9235748Z psmouse 32768 0 2024-06-26T07:09:23.9236134Z evdev 20480 3 2024-06-26T07:09:23.9236532Z button 16384 0 2024-06-26T07:09:23.9236930Z ena 139264 0 2024-06-26T07:09:23.9237610Z ptp 20480 1 ena 2024-06-26T07:09:23.9238041Z pps_core 20480 1 ptp 2024-06-26T07:09:23.9238469Z crc32c_intel 24576 0 2024-06-26T07:09:23.9238867Z autofs4 49152 2 2024-06-26T07:09:23.9239248Z + modinfo nvidia 2024-06-26T07:09:23.9239933Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:23.9240652Z alias: char-major-195-* 2024-06-26T07:09:23.9241059Z version: 550.54.15 2024-06-26T07:09:23.9241427Z supported: external 2024-06-26T07:09:23.9241794Z license: NVIDIA 2024-06-26T07:09:23.9242198Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:23.9242715Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:23.9243193Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:23.9243683Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:23.9244207Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:23.9244843Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:23.9245345Z depends: i2c-core,drm 2024-06-26T07:09:23.9245738Z retpoline: Y 2024-06-26T07:09:23.9246074Z name: nvidia 2024-06-26T07:09:23.9246641Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:23.9247320Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:23.9248005Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:23.9248670Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:23.9249135Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:23.9249599Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:23.9250078Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:23.9250535Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:23.9250988Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:23.9251539Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:23.9252139Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:23.9252646Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:23.9253096Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:23.9253539Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:23.9254130Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:23.9254688Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:23.9255286Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:23.9255872Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:23.9256507Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:23.9257134Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:23.9257759Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:23.9258383Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:23.9258902Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:23.9259459Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:23.9260021Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:23.9260544Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:23.9261033Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:23.9261541Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:23.9262133Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:23.9262635Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:23.9263169Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:23.9263707Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:23.9264212Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:23.9264727Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:23.9265253Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:23.9265988Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:23.9266526Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:23.9267029Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:23.9267470Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:23.9267961Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:23.9268457Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:23.9269001Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:23.9269515Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:23.9270008Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:23.9270493Z parm: rm_firmware_active:charp 2024-06-26T07:09:23.9270932Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:09:23.9271338Z ++ command -v nvidia-smi 2024-06-26T07:09:23.9271762Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:09:23.9272165Z + set +e 2024-06-26T07:09:23.9272711Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:09:25.9250707Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:09:25.9251365Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:25.9252268Z + '[' 0 -ne 0 ']' 2024-06-26T07:09:25.9252674Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:09:25.9253072Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:09:25.9253806Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:09:25.9254536Z + set -e 2024-06-26T07:09:25.9254855Z + '[' 1 -eq 0 ']' 2024-06-26T07:09:25.9255214Z + post_install_nvidia_driver_common 2024-06-26T07:09:25.9255897Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:09:25.9256580Z + sudo modprobe nvidia 2024-06-26T07:09:25.9352469Z + echo 'After installing NVIDIA driver' 2024-06-26T07:09:25.9353086Z + lspci 2024-06-26T07:09:25.9353421Z After installing NVIDIA driver 2024-06-26T07:09:25.9430886Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:25.9432391Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:25.9433787Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:25.9435144Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:25.9436296Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:25.9437868Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:25.9439021Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:25.9439876Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:25.9440535Z + lsmod 2024-06-26T07:09:25.9445294Z Module Size Used by 2024-06-26T07:09:25.9445922Z xt_nat 16384 0 2024-06-26T07:09:25.9446610Z ib_core 266240 0 2024-06-26T07:09:25.9447134Z nvidia_modeset 1339392 0 2024-06-26T07:09:25.9447583Z veth 16384 0 2024-06-26T07:09:25.9448120Z nvidia_uvm 4599808 0 2024-06-26T07:09:25.9448741Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:25.9449269Z drm 421888 1 nvidia 2024-06-26T07:09:25.9449827Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:25.9450465Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:25.9451016Z xt_conntrack 16384 1 2024-06-26T07:09:25.9451427Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:25.9451881Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:25.9452365Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:25.9452986Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:25.9453635Z xfrm_user 45056 1 2024-06-26T07:09:25.9454177Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:25.9454804Z iptable_nat 16384 1 2024-06-26T07:09:25.9455342Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:25.9455944Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:25.9456608Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:25.9457232Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:25.9458232Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:25.9459118Z xt_addrtype 16384 2 2024-06-26T07:09:25.9459522Z iptable_filter 16384 1 2024-06-26T07:09:25.9459943Z br_netfilter 24576 0 2024-06-26T07:09:25.9460504Z bridge 172032 1 br_netfilter 2024-06-26T07:09:25.9461120Z stp 16384 1 bridge 2024-06-26T07:09:25.9461559Z llc 16384 2 bridge,stp 2024-06-26T07:09:25.9462000Z overlay 86016 0 2024-06-26T07:09:25.9462458Z sunrpc 393216 1 2024-06-26T07:09:25.9462855Z dm_mirror 28672 0 2024-06-26T07:09:25.9463276Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:25.9463782Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:25.9464465Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:25.9464942Z dax 69632 1 dm_mod 2024-06-26T07:09:25.9465370Z crc32_pclmul 16384 0 2024-06-26T07:09:25.9465768Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:25.9466168Z pcbc 16384 0 2024-06-26T07:09:25.9466564Z aesni_intel 188416 0 2024-06-26T07:09:25.9466972Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:25.9467442Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:25.9467919Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:25.9468499Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:25.9469095Z mousedev 24576 0 2024-06-26T07:09:25.9469495Z psmouse 32768 0 2024-06-26T07:09:25.9469886Z evdev 20480 3 2024-06-26T07:09:25.9470272Z button 16384 0 2024-06-26T07:09:25.9470647Z ena 139264 0 2024-06-26T07:09:25.9471048Z ptp 20480 1 ena 2024-06-26T07:09:25.9471469Z pps_core 20480 1 ptp 2024-06-26T07:09:25.9471882Z crc32c_intel 24576 0 2024-06-26T07:09:25.9472274Z autofs4 49152 2 2024-06-26T07:09:25.9472654Z + modinfo nvidia 2024-06-26T07:09:25.9473326Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:25.9474094Z alias: char-major-195-* 2024-06-26T07:09:25.9474501Z version: 550.54.15 2024-06-26T07:09:25.9474979Z supported: external 2024-06-26T07:09:25.9475342Z license: NVIDIA 2024-06-26T07:09:25.9475740Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:25.9476244Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:25.9476724Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:25.9477214Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:25.9477991Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:25.9478516Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:25.9479024Z depends: i2c-core,drm 2024-06-26T07:09:25.9479406Z retpoline: Y 2024-06-26T07:09:25.9479739Z name: nvidia 2024-06-26T07:09:25.9480309Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:25.9480987Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:25.9481640Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:25.9482248Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:25.9482712Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:25.9483171Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:25.9483639Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:25.9484092Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:25.9484553Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:25.9485098Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:25.9485672Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:25.9486170Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:25.9486622Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:25.9487059Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:25.9487533Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:25.9488085Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:25.9488718Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:25.9489323Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:25.9489943Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:25.9490560Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:25.9491189Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:25.9491802Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:25.9492308Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:25.9492962Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:25.9493534Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:25.9494046Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:25.9494529Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:25.9495041Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:25.9495535Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:25.9495999Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:25.9496523Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:25.9497070Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:25.9497571Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:25.9498070Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:25.9498589Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:25.9499094Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:25.9499626Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:25.9500121Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:25.9500557Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:25.9501047Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:25.9501540Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:25.9502088Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:25.9502586Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:25.9503081Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:25.9503552Z parm: rm_firmware_active:charp 2024-06-26T07:09:25.9503973Z + set +e 2024-06-26T07:09:25.9504300Z + nvidia-smi 2024-06-26T07:09:27.5052422Z Wed Jun 26 07:09:27 2024 2024-06-26T07:09:27.5053182Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:27.5054023Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:09:27.5054858Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:27.5055663Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:09:27.5056568Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:09:27.5057295Z | | | MIG M. | 2024-06-26T07:09:27.5057861Z |=========================================+========================+======================| 2024-06-26T07:09:27.5146588Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:09:27.5147342Z | 0% 34C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:09:27.5148001Z | | | N/A | 2024-06-26T07:09:27.5148696Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:27.5149313Z 2024-06-26T07:09:27.5150020Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:27.5150674Z | Processes: | 2024-06-26T07:09:27.5151403Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:09:27.5152119Z | ID ID Usage | 2024-06-26T07:09:27.5152693Z |=========================================================================================| 2024-06-26T07:09:27.5153362Z | No running processes found | 2024-06-26T07:09:27.5154126Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:28.1193660Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:09:29.6762444Z NVIDIA A10G 2024-06-26T07:09:30.1126434Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:30.1127135Z + '[' 0 -eq 0 ']' 2024-06-26T07:09:30.1127761Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:09:30.1128367Z + set -e 2024-06-26T07:09:30.1128724Z INFO: Ignoring allowed status 0 2024-06-26T07:09:30.1130497Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:09:30.1132563Z + sudo yum install -y yum-utils 2024-06-26T07:09:30.4174050Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:31.9181694Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:09:31.9182492Z Nothing to do 2024-06-26T07:09:32.0662499Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:09:32.0663619Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:32.0665131Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:32.3986450Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:32.4263066Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:32.4264273Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:32.4265429Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:32.4379098Z + sudo yum install -y nvidia-docker2 2024-06-26T07:09:32.7403829Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:34.2453790Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:09:34.2471717Z Nothing to do 2024-06-26T07:09:34.3939624Z + sudo systemctl restart docker 2024-06-26T07:10:14.5895414Z Command completed after 1 attempt(s). 2024-06-26T07:10:14.5955452Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:14.5956244Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:14.5956956Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:10:14.5957837Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:14.5965667Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:14.5966191Z env: 2024-06-26T07:10:14.5966502Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:14.5966980Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:14.5967485Z ##[endgroup] 2024-06-26T07:10:14.8176460Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:14.8338269Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:10:14.8437578Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:10:14.9508401Z Prepare all required actions 2024-06-26T07:10:14.9508931Z Getting action download info 2024-06-26T07:10:15.0704871Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:10:15.4321024Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:10:15.5797856Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:10:15.5798368Z with: 2024-06-26T07:10:15.5798726Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:15.5799218Z s3-bucket: gha-artifacts 2024-06-26T07:10:15.5799588Z env: 2024-06-26T07:10:15.5799891Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:15.5800365Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:15.5800866Z ##[endgroup] 2024-06-26T07:10:15.5833565Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:15.5834035Z with: 2024-06-26T07:10:15.5834451Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:15.5835043Z s3-bucket: gha-artifacts 2024-06-26T07:10:15.5835425Z region: us-east-1 2024-06-26T07:10:15.5835761Z env: 2024-06-26T07:10:15.5836074Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:15.5836555Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:15.5837061Z ##[endgroup] 2024-06-26T07:10:15.9977228Z (node:11294) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:15.9977953Z 2024-06-26T07:10:15.9978228Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:15.9978977Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:15.9979997Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:16.0783688Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:10:16.0784805Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:23.5643982Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:23.5650597Z Artifact download has finished successfully 2024-06-26T07:10:23.5810303Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:10:23.5810770Z unzip -o artifacts.zip 2024-06-26T07:10:23.5818589Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:23.5819111Z env: 2024-06-26T07:10:23.5819412Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:23.5819891Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:23.5820394Z ##[endgroup] 2024-06-26T07:10:23.5848193Z Archive: artifacts.zip 2024-06-26T07:10:23.5849999Z creating: dist/ 2024-06-26T07:10:25.5549567Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:25.5551334Z creating: build/custom_test_artifacts/ 2024-06-26T07:10:25.5552590Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:10:25.5554065Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:10:25.5555943Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:25.5557219Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:25.5558471Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:25.5559444Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:25.5560575Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:25.5561804Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:25.5562953Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:25.5564284Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:25.5565380Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:25.5566391Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:25.5567611Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:25.5569023Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:25.5570262Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:25.5571821Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:25.5573088Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:25.5574270Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:25.5575313Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:25.5576328Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:25.5610843Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:25.5648880Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:25.5650350Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:25.5692750Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:25.5694731Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:25.5696719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:25.5698704Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:25.5700557Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:25.5701957Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:25.5703367Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:25.5704759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:25.5706115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:25.5707395Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:25.5708635Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:25.5709871Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:25.5711112Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:25.5712308Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:25.5713534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:25.5758733Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:25.5816200Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:25.5817835Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:25.5819170Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:25.5820352Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:25.5821391Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:25.5822331Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:10:25.5823364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:25.5824517Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:10:25.5825638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:10:25.5826739Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:10:25.5827821Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:25.5828919Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:10:25.5829998Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:25.5831078Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:10:25.5832150Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:10:25.5840775Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:10:25.5967958Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:10:25.5969354Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:10:25.5970805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:25.5972417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:10:25.5973587Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:10:25.5974683Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:10:25.5975817Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:25.5976964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:10:25.5978110Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:25.5979251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:10:25.5980369Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:10:25.5991696Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:10:25.6070657Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:10:25.6072186Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:25.6073567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:25.6074770Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:10:25.6075949Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:10:25.6077143Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:25.6078497Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:10:25.6079356Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:10:25.6080147Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:10:25.6080965Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:10:25.6184858Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:10:25.6244966Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:10:25.6245896Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:10:25.6246613Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:10:25.6247441Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:25.6252341Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:25.6253421Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:25.6254359Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:25.6255609Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:25.6256828Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:25.6257944Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:25.6259229Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:25.6260445Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:25.6261619Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:25.6263013Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:25.6264712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:25.6266159Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:25.6267300Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:25.6268457Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:25.6269610Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:25.6270634Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:25.6271631Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:25.6304379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:25.6343076Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:25.6345615Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:25.6385765Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:25.6387412Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:25.6389415Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:25.6391438Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:25.6393238Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:25.6394946Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:25.6396324Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:25.6397928Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:25.6400162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:25.6401483Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:25.6402722Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:25.6403937Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:25.6405185Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:25.6406377Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:25.6407591Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:25.6452605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:25.6511019Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:25.6513410Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:25.6515423Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:25.6516566Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:25.6517730Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:25.6518699Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:10:25.6519767Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:10:25.6520959Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:10:25.6522113Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:10:25.6523187Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:10:25.6524307Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:10:25.6525422Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:10:25.6526555Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:10:25.6527666Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:10:25.6528763Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:10:25.6535105Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:10:25.6596463Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:10:25.6598006Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:25.6599110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:25.6600083Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:10:25.6600989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:10:25.6601899Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:25.6602807Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:10:25.6603650Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:10:25.6604439Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:10:25.6605265Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:10:25.6650416Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:10:25.6651190Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:10:25.6651954Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:10:25.6652852Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:25.6657741Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:25.6658745Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:25.6659745Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:25.6660802Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:25.6661844Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:25.6678861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:25.6680053Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:25.6681254Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:25.6682306Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:25.6683507Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:25.6684740Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:25.6685928Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:25.6687117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:25.6688321Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:25.6689532Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:25.6690633Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:25.6691694Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:25.6708396Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:25.6746095Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:25.6747753Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:25.6789848Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:25.6791326Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:25.6792813Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:25.6794335Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:25.6795868Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:25.6807603Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:25.6809059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:25.6810489Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:25.6811884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:25.6813226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:25.6814520Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:25.6815787Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:25.6817071Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:25.6818309Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:25.6819574Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:25.6873802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:25.6932491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:25.6933753Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:25.6934801Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:25.6935730Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:25.6936702Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:25.6937724Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:10:25.6938867Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:25.6940117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:10:25.6941326Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:10:25.6942469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:10:25.6943638Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:25.6944932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:10:25.6946180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:25.6947369Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:10:25.6948543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:10:25.6949794Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:10:25.7065655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:10:25.7067479Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:10:25.7069229Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:25.7071188Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:10:25.7073041Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:10:25.7074880Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:10:25.7076718Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:25.7078827Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:10:25.7080625Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:25.7082410Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:10:25.7084194Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:10:25.7093546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:10:25.7147365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:10:25.7149447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:25.7151134Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:25.7152605Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:10:25.7154040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:10:25.7155574Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:25.7157045Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:10:25.7158649Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:10:25.7159833Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:10:25.7161070Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:10:25.7256860Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:10:25.7298774Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:10:25.7299723Z creating: build/lib/ 2024-06-26T07:10:25.7384339Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:10:25.7823059Z inflating: build/lib/libprotobuf.a 2024-06-26T07:10:25.7832559Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:10:25.7841099Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:10:25.7848996Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:10:25.7849734Z inflating: build/lib/libclog.a 2024-06-26T07:10:25.7867535Z inflating: build/lib/libnnpack.a 2024-06-26T07:10:25.7869906Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:10:25.7932670Z inflating: build/lib/libgtest.a 2024-06-26T07:10:25.8005061Z inflating: build/lib/libbenchmark.a 2024-06-26T07:10:25.8066923Z inflating: build/lib/libasmjit.a 2024-06-26T07:10:25.8074377Z inflating: build/lib/libittnotify.a 2024-06-26T07:10:25.8101671Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:10:25.8224026Z inflating: build/lib/libgloo.a 2024-06-26T07:10:25.8245631Z inflating: build/lib/libfmt.a 2024-06-26T07:10:25.8334600Z inflating: build/lib/libc10.so 2024-06-26T07:10:25.8336272Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:10:25.8337014Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:10:25.8338593Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:10:25.8358093Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:10:25.8375666Z inflating: build/lib/libgmock.a 2024-06-26T07:10:25.8376351Z inflating: build/lib/libgtest_main.a 2024-06-26T07:10:25.8377049Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:10:25.8863943Z inflating: build/lib/libprotoc.a 2024-06-26T07:10:25.9233073Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:10:26.8954997Z inflating: build/lib/libdnnl.a 2024-06-26T07:10:26.9505728Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:10:26.9558917Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:10:26.9559615Z inflating: build/lib/libgmock_main.a 2024-06-26T07:10:27.0760950Z inflating: build/lib/libfbgemm.a 2024-06-26T07:10:27.1006305Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:10:27.1503526Z inflating: build/lib/libkineto.a 2024-06-26T07:10:27.1681666Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:10:27.1722585Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:10:27.2386468Z inflating: build/lib/libonnx.a 2024-06-26T07:10:29.5675280Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:10:29.5679459Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:10:29.5683488Z inflating: build/lib/libshm.so 2024-06-26T07:10:31.5538514Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:10:31.5539464Z inflating: build/lib/libtorch.so 2024-06-26T07:10:31.5541922Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:10:32.3491784Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:10:32.5391521Z inflating: build/lib/libtorch_python.so 2024-06-26T07:10:32.5457962Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:10:32.5478401Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:10:32.5503218Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:10:32.5538229Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:10:32.5538931Z creating: build/bin/ 2024-06-26T07:10:32.5587842Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:10:32.5638334Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:10:32.5687412Z inflating: build/bin/c10_Device_test 2024-06-26T07:10:32.5745178Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:10:32.5797448Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:10:32.5845372Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:10:32.5895091Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:10:32.5948329Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:10:32.6002812Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:10:32.6057655Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:10:32.6126901Z inflating: build/bin/c10_cow_test 2024-06-26T07:10:32.6178811Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:10:32.6226553Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:10:32.6275421Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:10:32.6324814Z inflating: build/bin/c10_Half_test 2024-06-26T07:10:32.6379607Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:10:32.6433471Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:10:32.6482700Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:10:32.6536786Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:10:32.6587113Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:10:32.6637113Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:10:32.6684800Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:10:32.6734817Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:10:32.6788562Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:10:32.6838609Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:10:32.6892163Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:10:32.6945684Z inflating: build/bin/c10_complex_test 2024-06-26T07:10:32.6996981Z inflating: build/bin/c10_exception_test 2024-06-26T07:10:32.7045305Z inflating: build/bin/c10_flags_test 2024-06-26T07:10:32.7093743Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:10:32.7251879Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:10:32.7301600Z inflating: build/bin/c10_irange_test 2024-06-26T07:10:32.7352710Z inflating: build/bin/c10_lazy_test 2024-06-26T07:10:32.7408424Z inflating: build/bin/c10_logging_test 2024-06-26T07:10:32.7480805Z inflating: build/bin/c10_optional_test 2024-06-26T07:10:32.7541067Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:10:32.7593539Z inflating: build/bin/c10_registry_test 2024-06-26T07:10:32.7739132Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:10:32.7789526Z inflating: build/bin/c10_ssize_test 2024-06-26T07:10:32.7839617Z inflating: build/bin/c10_string_util_test 2024-06-26T07:10:32.7896288Z inflating: build/bin/c10_string_view_test 2024-06-26T07:10:32.7945399Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:10:32.7992069Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:10:32.8046536Z inflating: build/bin/c10_typeid_test 2024-06-26T07:10:32.8480159Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:10:32.8912812Z inflating: build/bin/protoc 2024-06-26T07:10:32.8964488Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:10:32.9015639Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:10:32.9066502Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:10:32.9116739Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:10:32.9167751Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:10:32.9218853Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:10:32.9266105Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:10:32.9317070Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:10:32.9634867Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:10:32.9966187Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:10:33.0312164Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:10:33.0363250Z inflating: build/bin/FileStoreTest 2024-06-26T07:10:33.0417153Z inflating: build/bin/TCPStoreTest 2024-06-26T07:10:33.0468740Z inflating: build/bin/HashStoreTest 2024-06-26T07:10:33.0481945Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:10:33.0535011Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:10:33.0538951Z inflating: build/bin/torch_shm_manager 2024-06-26T07:10:33.0541588Z inflating: build/bin/example_allreduce 2024-06-26T07:10:33.0595747Z inflating: build/bin/test_dist_autograd 2024-06-26T07:10:33.0663855Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:10:33.0665447Z inflating: build/bin/parallel_benchmark 2024-06-26T07:10:33.0731225Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:10:33.0739783Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:10:33.1072930Z inflating: build/bin/test_lazy 2024-06-26T07:10:33.2215029Z inflating: build/bin/test_api 2024-06-26T07:10:33.2287121Z inflating: build/bin/Dict_test 2024-06-26T07:10:33.2336771Z inflating: build/bin/Dimname_test 2024-06-26T07:10:33.2399387Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:10:33.2455028Z inflating: build/bin/NamedTensor_test 2024-06-26T07:10:33.2512263Z inflating: build/bin/apply_utils_test 2024-06-26T07:10:33.2569485Z inflating: build/bin/atest 2024-06-26T07:10:33.2630568Z inflating: build/bin/basic 2024-06-26T07:10:33.2683642Z inflating: build/bin/broadcast_test 2024-06-26T07:10:33.2732972Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:10:33.2789397Z inflating: build/bin/cpu_generator_test 2024-06-26T07:10:33.2841054Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:10:33.2931638Z inflating: build/bin/cpu_rng_test 2024-06-26T07:10:33.2980491Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:10:33.3028932Z inflating: build/bin/dlconvertor_test 2024-06-26T07:10:33.3085045Z inflating: build/bin/extension_backend_test 2024-06-26T07:10:33.3138441Z inflating: build/bin/half_test 2024-06-26T07:10:33.3231050Z inflating: build/bin/ivalue_test 2024-06-26T07:10:33.3279442Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:10:33.3332123Z inflating: build/bin/math_kernel_test 2024-06-26T07:10:33.3385122Z inflating: build/bin/memory_format_test 2024-06-26T07:10:33.3435900Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:10:33.3487283Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:10:33.3541844Z inflating: build/bin/native_test 2024-06-26T07:10:33.3590165Z inflating: build/bin/operator_name_test 2024-06-26T07:10:33.3640198Z inflating: build/bin/operators_test 2024-06-26T07:10:33.3690245Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:10:33.3755707Z inflating: build/bin/pow_test 2024-06-26T07:10:33.3810972Z inflating: build/bin/quantized_test 2024-06-26T07:10:33.3859236Z inflating: build/bin/reduce_ops_test 2024-06-26T07:10:33.3909320Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:10:33.3963559Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:10:33.4020545Z inflating: build/bin/scalar_test 2024-06-26T07:10:33.4070973Z inflating: build/bin/StorageUtils_test 2024-06-26T07:10:33.4121303Z inflating: build/bin/stride_properties_test 2024-06-26T07:10:33.4197206Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:10:33.4249604Z inflating: build/bin/test_parallel 2024-06-26T07:10:33.4252320Z inflating: build/bin/thread_init_test 2024-06-26T07:10:33.4306881Z inflating: build/bin/type_ptr_test 2024-06-26T07:10:33.4364333Z inflating: build/bin/type_test 2024-06-26T07:10:33.4414796Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:10:33.4416204Z inflating: build/bin/verify_api_visibility 2024-06-26T07:10:33.4483370Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:10:33.4533153Z inflating: build/bin/weakref_test 2024-06-26T07:10:33.4583003Z inflating: build/bin/wrapdim_test 2024-06-26T07:10:33.4633193Z inflating: build/bin/xla_tensor_test 2024-06-26T07:10:33.4691000Z inflating: build/bin/IListRef_test 2024-06-26T07:10:33.4793816Z inflating: build/bin/List_test 2024-06-26T07:10:33.4858805Z inflating: build/bin/KernelFunction_test 2024-06-26T07:10:33.4976003Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:10:33.5069467Z inflating: build/bin/kernel_function_test 2024-06-26T07:10:33.5193451Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:10:33.5293729Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:10:33.5353071Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:10:33.5445693Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:10:33.5495092Z inflating: build/bin/CppSignature_test 2024-06-26T07:10:33.5549786Z inflating: build/bin/backend_fallback_test 2024-06-26T07:10:33.5596677Z inflating: build/bin/op_allowlist_test 2024-06-26T07:10:33.5658517Z inflating: build/bin/inline_container_test 2024-06-26T07:10:33.5950082Z inflating: build/bin/op_registration_test 2024-06-26T07:10:33.6001527Z inflating: build/bin/cuda_apply_test 2024-06-26T07:10:33.6050618Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:10:33.6107661Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:10:33.6160718Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:10:33.6228527Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:10:33.6285403Z inflating: build/bin/cuda_complex_test 2024-06-26T07:10:33.6333432Z inflating: build/bin/cuda_device_test 2024-06-26T07:10:33.6388998Z inflating: build/bin/cuda_cub_test 2024-06-26T07:10:33.6438228Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:10:33.6501321Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:10:33.6555619Z inflating: build/bin/cuda_generator_test 2024-06-26T07:10:33.6603923Z inflating: build/bin/cuda_half_test 2024-06-26T07:10:33.6653351Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:10:33.6700896Z inflating: build/bin/cuda_optional_test 2024-06-26T07:10:33.6750985Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:10:33.6802124Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:10:33.6850417Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:10:33.6909014Z inflating: build/bin/cuda_stream_test 2024-06-26T07:10:33.6956848Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:10:33.7007279Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:10:33.7021662Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:10:33.7085312Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:10:33.7140806Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:10:33.7202334Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:10:33.7263313Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:10:33.8068548Z inflating: build/bin/test_tensorexpr 2024-06-26T07:10:33.8628628Z inflating: build/bin/test_jit 2024-06-26T07:10:33.8629551Z creating: .additional_ci_files/ 2024-06-26T07:10:33.8682466Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:10:33.8892954Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:10:33.8921006Z ##[group]Run rm artifacts.zip 2024-06-26T07:10:33.8921421Z rm artifacts.zip 2024-06-26T07:10:33.8929142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:33.8929659Z env: 2024-06-26T07:10:33.8929973Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.8930449Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.8930950Z ##[endgroup] 2024-06-26T07:10:33.9637219Z ##[group]Run df -H 2024-06-26T07:10:33.9637845Z df -H 2024-06-26T07:10:33.9645753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:33.9646283Z env: 2024-06-26T07:10:33.9646585Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.9647064Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.9647568Z ##[endgroup] 2024-06-26T07:10:33.9673469Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:10:33.9674131Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:10:33.9674991Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:10:33.9675689Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:10:33.9676374Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:10:33.9676910Z /dev/nvme0n1p1 162G 47G 115G 29% / 2024-06-26T07:10:33.9714088Z Prepare all required actions 2024-06-26T07:10:33.9714549Z Getting action download info 2024-06-26T07:10:34.1068981Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:10:34.1069458Z with: 2024-06-26T07:10:34.1069740Z env: 2024-06-26T07:10:34.1070045Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:34.1070516Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:34.1071022Z ##[endgroup] 2024-06-26T07:10:34.1102427Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:34.1102890Z with: 2024-06-26T07:10:34.1103177Z name: td_results 2024-06-26T07:10:34.1103517Z s3-bucket: gha-artifacts 2024-06-26T07:10:34.1103890Z region: us-east-1 2024-06-26T07:10:34.1104210Z env: 2024-06-26T07:10:34.1104500Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:34.1104971Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:34.1105475Z ##[endgroup] 2024-06-26T07:10:34.5204180Z (node:11315) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:34.5204873Z 2024-06-26T07:10:34.5205140Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:34.5205881Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:34.5206752Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:34.5835217Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:10:34.5841585Z Artifact download has finished successfully 2024-06-26T07:10:34.5970791Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:10:34.5971293Z mkdir -p .additional_ci_files 2024-06-26T07:10:34.5971870Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:10:34.5979694Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:34.5980201Z env: 2024-06-26T07:10:34.5980506Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:34.5980990Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:34.5981492Z ##[endgroup] 2024-06-26T07:10:34.6028844Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:10:34.6040505Z ##[error]Process completed with exit code 1. 2024-06-26T07:10:34.6071927Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:10:34.6072429Z .github/scripts/parse_ref.py 2024-06-26T07:10:34.6080076Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:34.6080447Z env: 2024-06-26T07:10:34.6080755Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:34.6081226Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:34.6081733Z ##[endgroup] 2024-06-26T07:10:34.6307158Z Prepare all required actions 2024-06-26T07:10:34.6344457Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:10:34.6344936Z with: 2024-06-26T07:10:34.6345517Z github-token: *** 2024-06-26T07:10:34.6345851Z env: 2024-06-26T07:10:34.6346148Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:34.6346624Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:34.6347125Z ##[endgroup] 2024-06-26T07:10:34.6362992Z ##[group]Run set -eux 2024-06-26T07:10:34.6363355Z set -eux 2024-06-26T07:10:34.6363956Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:10:34.6371883Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:34.6372435Z env: 2024-06-26T07:10:34.6372730Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:34.6373216Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:34.6373920Z GITHUB_TOKEN: *** 2024-06-26T07:10:34.6374252Z ##[endgroup] 2024-06-26T07:10:34.6393708Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0ff3df5957bf06960 2024-06-26T07:10:35.9416636Z setting job-id=26688911259 2024-06-26T07:10:35.9417500Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:35.9609771Z Prepare all required actions 2024-06-26T07:10:35.9610233Z Getting action download info 2024-06-26T07:10:36.0703154Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:10:36.0703629Z with: 2024-06-26T07:10:36.0704106Z github-token: *** 2024-06-26T07:10:36.0712083Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:10:36.0720928Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:36.0721632Z env: 2024-06-26T07:10:36.0722082Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.0722557Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.0723050Z ##[endgroup] 2024-06-26T07:10:36.0764327Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:10:36.0764901Z with: 2024-06-26T07:10:36.0765195Z shell: bash 2024-06-26T07:10:36.0765522Z timeout_minutes: 10 2024-06-26T07:10:36.0765877Z max_attempts: 5 2024-06-26T07:10:36.0766211Z retry_wait_seconds: 30 2024-06-26T07:10:36.0767336Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:36.0768510Z polling_interval_seconds: 1 2024-06-26T07:10:36.0768910Z warning_on_retry: true 2024-06-26T07:10:36.0769280Z continue_on_error: false 2024-06-26T07:10:36.0769644Z env: 2024-06-26T07:10:36.0769950Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.0770418Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.0771081Z GITHUB_TOKEN: *** 2024-06-26T07:10:36.0771423Z ##[endgroup] 2024-06-26T07:10:36.1266605Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:36.3371166Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:36.3521113Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:10:36.3662402Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:10:36.3670989Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:10:36.3683588Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:10:36.3692845Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:10:36.3884881Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:10:37.1260613Z Command completed after 1 attempt(s). 2024-06-26T07:10:37.1300908Z ##[group]Run set -x 2024-06-26T07:10:37.1301267Z set -x 2024-06-26T07:10:37.1301591Z  2024-06-26T07:10:37.1302175Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:37.1302869Z # in runner workspace 2024-06-26T07:10:37.1303421Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:10:37.1311222Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:37.1311743Z env: 2024-06-26T07:10:37.1312044Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:37.1312517Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:37.1313032Z ##[endgroup] 2024-06-26T07:10:37.1334600Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:10:37.1535047Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:37.1535620Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:37.1536107Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:10:37.1536523Z  2024-06-26T07:10:37.1537104Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:37.1537795Z # in runner workspace 2024-06-26T07:10:37.1538394Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:10:37.1539053Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:10:37.1539526Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:10:37.1547750Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:10:37.1573806Z  --selected-test-configs "" \ 2024-06-26T07:10:37.1574291Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:10:37.1574728Z  --tag "${TAG}" \ 2024-06-26T07:10:37.1575132Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:10:37.1575588Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:10:37.1576033Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:10:37.1583665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:37.1584175Z env: 2024-06-26T07:10:37.1584472Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:37.1584948Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:37.1585637Z GITHUB_TOKEN: *** 2024-06-26T07:10:37.1586271Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:37.1586975Z PR_NUMBER: 2024-06-26T07:10:37.1587304Z TAG: ciflow/inductor/129470 2024-06-26T07:10:37.1587696Z EVENT_NAME: push 2024-06-26T07:10:37.1588034Z SCHEDULE: 2024-06-26T07:10:37.1588338Z HEAD_BRANCH: 2024-06-26T07:10:37.1588660Z ##[endgroup] 2024-06-26T07:10:37.1608540Z Workflow: inductor 2024-06-26T07:10:37.1609556Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:37.3926978Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:10:37.5327000Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:10:37.5327487Z echo "Filtered matrix:" 2024-06-26T07:10:37.5335629Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:10:37.5344148Z  2024-06-26T07:10:37.5344531Z echo 2024-06-26T07:10:37.5344939Z echo "Is the current job unstable? False" 2024-06-26T07:10:37.5345413Z  2024-06-26T07:10:37.5345711Z echo 2024-06-26T07:10:37.5346095Z echo "Is keep-going label set? False" 2024-06-26T07:10:37.5346565Z  2024-06-26T07:10:37.5346859Z echo 2024-06-26T07:10:37.5347197Z echo "Renabled issues? " 2024-06-26T07:10:37.5355097Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:37.5355626Z env: 2024-06-26T07:10:37.5355925Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:37.5356397Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:37.5356898Z ##[endgroup] 2024-06-26T07:10:37.5377267Z Filtered matrix: 2024-06-26T07:10:37.5387654Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:10:37.5395998Z 2024-06-26T07:10:37.5396167Z Is the current job unstable? False 2024-06-26T07:10:37.5396454Z 2024-06-26T07:10:37.5396685Z Is keep-going label set? False 2024-06-26T07:10:37.5396957Z 2024-06-26T07:10:37.5397096Z Renabled issues? 2024-06-26T07:10:37.5438515Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:37.5439231Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:37.5446977Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:37.5447487Z env: 2024-06-26T07:10:37.5447794Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:37.5448269Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:37.5448784Z JOB_TIMEOUT: 240 2024-06-26T07:10:37.5449111Z ##[endgroup] 2024-06-26T07:10:37.5521186Z ##[group]Run set -x 2024-06-26T07:10:37.5521600Z set -x 2024-06-26T07:10:37.5521929Z  2024-06-26T07:10:37.5522327Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:10:37.5522889Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:10:37.5523474Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:10:37.5524010Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:10:37.5524447Z else 2024-06-26T07:10:37.5524810Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:37.5525253Z fi 2024-06-26T07:10:37.5525559Z  2024-06-26T07:10:37.5526047Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:10:37.5526813Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:10:37.5527507Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:10:37.5528114Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:10:37.5528617Z container_name=$(docker run \ 2024-06-26T07:10:37.5529094Z  ${GPU_FLAG:-} \ 2024-06-26T07:10:37.5529497Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:10:37.5529919Z  -e PR_NUMBER \ 2024-06-26T07:10:37.5530300Z  -e GITHUB_ACTIONS \ 2024-06-26T07:10:37.5530717Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:10:37.5531149Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:10:37.5531558Z  -e GITHUB_JOB \ 2024-06-26T07:10:37.5531938Z  -e GITHUB_RUN_ID \ 2024-06-26T07:10:37.5532351Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:10:37.5532778Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:10:37.5533192Z  -e JOB_ID \ 2024-06-26T07:10:37.5533559Z  -e JOB_NAME \ 2024-06-26T07:10:37.5533932Z  -e BASE_SHA \ 2024-06-26T07:10:37.5534302Z  -e BRANCH \ 2024-06-26T07:10:37.5534654Z  -e SHA1 \ 2024-06-26T07:10:37.5535028Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:10:37.5535458Z  -e IN_WHEEL_TEST \ 2024-06-26T07:10:37.5535860Z  -e SHARD_NUMBER \ 2024-06-26T07:10:37.5536252Z  -e TEST_CONFIG \ 2024-06-26T07:10:37.5536654Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:10:37.5537070Z  -e REENABLED_ISSUES \ 2024-06-26T07:10:37.5537508Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:10:37.5537951Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:10:37.5538375Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:10:37.5538780Z  -e NO_TD \ 2024-06-26T07:10:37.5539151Z  -e TD_DISTRIBUTED \ 2024-06-26T07:10:37.5539552Z  -e PR_LABELS \ 2024-06-26T07:10:37.5539984Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:10:37.5540462Z  -e SCCACHE_BUCKET \ 2024-06-26T07:10:37.5540877Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:10:37.5541311Z  -e XLA_CUDA \ 2024-06-26T07:10:37.5541743Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:10:37.5542280Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:10:37.5542818Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:10:37.5543465Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:10:37.5543955Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:10:37.5544400Z  -e DASHBOARD_TAG \ 2024-06-26T07:10:37.5544893Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:10:37.5545470Z  --security-opt seccomp=unconfined \ 2024-06-26T07:10:37.5545962Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:10:37.5546383Z  --ipc=host \ 2024-06-26T07:10:37.5546760Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:10:37.5547183Z  --tty \ 2024-06-26T07:10:37.5547528Z  --detach \ 2024-06-26T07:10:37.5547912Z  --name="${container_name}" \ 2024-06-26T07:10:37.5548347Z  --user jenkins \ 2024-06-26T07:10:37.5548924Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:10:37.5549513Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:10:37.5549982Z  "${DOCKER_IMAGE}" 2024-06-26T07:10:37.5550354Z ) 2024-06-26T07:10:37.5550787Z # Propagate download.pytorch.org IP to container 2024-06-26T07:10:37.5551723Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:10:37.5552704Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:10:37.5553624Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:10:37.5561313Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:37.5561692Z env: 2024-06-26T07:10:37.5561996Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:37.5562468Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:37.5563089Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:37.5563614Z PR_NUMBER: 2024-06-26T07:10:37.5563964Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:10:37.5564409Z GITHUB_WORKFLOW: inductor 2024-06-26T07:10:37.5564783Z GITHUB_JOB: test 2024-06-26T07:10:37.5565125Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:10:37.5565508Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:10:37.5565876Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:10:37.5566236Z JOB_ID: 26688911259 2024-06-26T07:10:37.5566875Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:37.5567572Z BRANCH: 2024-06-26T07:10:37.5567928Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:37.5568462Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:37.5568951Z TEST_CONFIG: inductor_timm 2024-06-26T07:10:37.5569333Z SHARD_NUMBER: 2 2024-06-26T07:10:37.5569661Z NUM_TEST_SHARDS: 2 2024-06-26T07:10:37.5570008Z REENABLED_ISSUES: 2024-06-26T07:10:37.5570373Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:10:37.5570789Z VERBOSE_TEST_LOGS: False 2024-06-26T07:10:37.5571162Z NO_TEST_TIMEOUT: False 2024-06-26T07:10:37.5571527Z NO_TD: False 2024-06-26T07:10:37.5571857Z TD_DISTRIBUTED: False 2024-06-26T07:10:37.5572295Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:10:37.5572804Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:10:37.5573206Z SHM_SIZE: 2g 2024-06-26T07:10:37.5574301Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:37.5575473Z XLA_CUDA: 2024-06-26T07:10:37.5575968Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:37.5576604Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:10:37.5577060Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:10:37.5577488Z DASHBOARD_TAG: 2024-06-26T07:10:37.5578011Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:10:37.5578402Z ##[endgroup] 2024-06-26T07:10:37.5601522Z + [[ inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:10:37.5602220Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:10:37.5602911Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:37.5608744Z +++ nproc --ignore=2 2024-06-26T07:10:37.5621008Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:48.7972770Z + container_name=60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T07:10:48.7975205Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:10:48.7976462Z + docker exec -i 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:10:48.8559725Z + echo DOCKER_CONTAINER_ID=60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T07:10:48.8562204Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:48.8564276Z + docker exec -t 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:10:49.2499440Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:10:49.5644983Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:10:49.5649246Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:10:49.5653228Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:10:49.5655596Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:10:49.5659150Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:10:49.5662113Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:10:49.5688158Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:10:49.5747754Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:10:49.6177153Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:10:49.6335641Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:10:50.9434564Z Installing collected packages: torch 2024-06-26T07:10:59.6189612Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:10:59.6190967Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:10:59.6191657Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:10:59.6982321Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:10:59.6986764Z + source .ci/pytorch/common.sh 2024-06-26T07:10:59.6988316Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:10:59.6993191Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:10:59.6994859Z +++ declare -f -t trap_add 2024-06-26T07:10:59.7001171Z ++ set -ex 2024-06-26T07:10:59.7001856Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:10:59.7002435Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:10:59.7004400Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:10:59.7005312Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:10:59.7014421Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:10:59.7014983Z + trap_add cleanup_workspace EXIT 2024-06-26T07:10:59.7015593Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:10:59.7016168Z + shift 2024-06-26T07:10:59.7016612Z + for trap_add_name in "$@" 2024-06-26T07:10:59.7020353Z +++ trap -p EXIT 2024-06-26T07:10:59.7022343Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:10:59.7022746Z +++ extract_trap_cmd 2024-06-26T07:10:59.7023201Z +++ printf '%s\n' '' 2024-06-26T07:10:59.7023624Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:10:59.7025191Z + trap -- ' 2024-06-26T07:10:59.7025620Z cleanup_workspace' EXIT 2024-06-26T07:10:59.7026126Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:11:00.1175256Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:11:00.1186643Z + echo 'Environment variables:' 2024-06-26T07:11:00.1187171Z Environment variables: 2024-06-26T07:11:00.1187530Z + env 2024-06-26T07:11:00.1192060Z INSTALLED_DB=yes 2024-06-26T07:11:00.1192724Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:00.1193309Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:00.1193896Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:00.1194855Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:00.1195724Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:00.1198720Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:00.1200057Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:00.1200923Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:00.1201640Z HOSTNAME=60a3066c951d 2024-06-26T07:11:00.1202862Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1204053Z GITHUB_ACTION=__self 2024-06-26T07:11:00.1204612Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:00.1210445Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:00.1216501Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:00.1217301Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:00.1218094Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:11:00.1218647Z TEST_CONFIG=inductor_timm 2024-06-26T07:11:00.1219239Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:00.1220021Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:00.1220744Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:00.1221419Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:00.1222110Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:00.1222804Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:00.1223589Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:00.1224366Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:00.1224916Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:00.1225564Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:00.1226159Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1226869Z XLA_CUDA= 2024-06-26T07:11:00.1227821Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:00.1229783Z *** 2024-06-26T07:11:00.1230120Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:00.1230556Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:00.1230947Z GITHUB_ACTIONS=true 2024-06-26T07:11:00.1231305Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:00.1231837Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:00.1232396Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:00.1232886Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1233410Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:00.1233906Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1234828Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:00.1235613Z UCC_HOME=/usr 2024-06-26T07:11:00.1235977Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:00.1236395Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:00.1236762Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:00.1237249Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:00.1238196Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:00.1238701Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:00.1239103Z SHARD_NUMBER=2 2024-06-26T07:11:00.1239445Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:00.1239816Z HOME=/var/lib/jenkins 2024-06-26T07:11:00.1240207Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:00.1240672Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:00.1241170Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:00.1241668Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:11:00.1242064Z CUDA_VERSION=12.1.1 2024-06-26T07:11:00.1242508Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:00.1242970Z NUM_TEST_SHARDS=2 2024-06-26T07:11:00.1243294Z UCX_HOME=/usr 2024-06-26T07:11:00.1243826Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:00.1244925Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1246167Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:00.1247345Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1248456Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:00.1249124Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:00.1249480Z DASHBOARD_TAG= 2024-06-26T07:11:00.1249802Z GITHUB_RUN_ID=9673646808 2024-06-26T07:11:00.1250278Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:00.1250831Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:00.1251827Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1252737Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:00.1253170Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:00.1253580Z PR_NUMBER= 2024-06-26T07:11:00.1253885Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:00.1254401Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:00.1254870Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:00.1255402Z TERM=xterm 2024-06-26T07:11:00.1255754Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:00.1256192Z INSTALLED_VISION=yes 2024-06-26T07:11:00.1256531Z BRANCH= 2024-06-26T07:11:00.1256845Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:00.1257263Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:00.1257709Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:00.1258473Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:00.1259232Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:00.1259741Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:00.1260232Z REENABLED_ISSUES= 2024-06-26T07:11:00.1260554Z SHLVL=1 2024-06-26T07:11:00.1260838Z MAX_JOBS=14 2024-06-26T07:11:00.1261257Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:00.1261635Z NVARCH=x86_64 2024-06-26T07:11:00.1261965Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:00.1262438Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1262996Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:00.1263503Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:00.1264172Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:00.1264764Z GITHUB_JOB=test 2024-06-26T07:11:00.1265178Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:00.1265753Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:00.1266388Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:00.1266746Z TD_DISTRIBUTED=False 2024-06-26T07:11:00.1267150Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:00.1267609Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:00.1268044Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:00.1268442Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:00.1268818Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:00.1269185Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:00.1270245Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:00.1271319Z GITHUB_BASE_REF= 2024-06-26T07:11:00.1271669Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:00.1272058Z CI=true 2024-06-26T07:11:00.1272395Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:00.1272832Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:00.1273229Z JOB_ID=26688911259 2024-06-26T07:11:00.1273563Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:00.1273915Z GITHUB_HEAD_REF= 2024-06-26T07:11:00.1274243Z GITHUB_ACTION_REF= 2024-06-26T07:11:00.1274776Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:00.1275314Z GITHUB_WORKFLOW=inductor 2024-06-26T07:11:00.1275713Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:00.1276629Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1277741Z NO_TD=False 2024-06-26T07:11:00.1278068Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:00.1278460Z _=/usr/bin/env 2024-06-26T07:11:00.1278963Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:11:00.1344394Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:11:00.1345738Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:11:00.1346966Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:11:00.1347888Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:11:00.1348514Z + BUILD_DIR=build 2024-06-26T07:11:00.1348863Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:11:00.1349279Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:11:00.1349647Z + SHARD_NUMBER=2 2024-06-26T07:11:00.1349981Z + NUM_TEST_SHARDS=2 2024-06-26T07:11:00.1350329Z + export VALGRIND=ON 2024-06-26T07:11:00.1350676Z + VALGRIND=ON 2024-06-26T07:11:00.1351173Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:11:00.1351829Z + [[ 0 == \1 ]] 2024-06-26T07:11:00.1352157Z + [[ False == \1 ]] 2024-06-26T07:11:00.1352650Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:11:00.1353193Z ++ realpath build/custom_test_artifacts 2024-06-26T07:11:00.1357943Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:11:00.1358950Z + [[ -n '' ]] 2024-06-26T07:11:00.1359495Z + echo 'Environment variables' 2024-06-26T07:11:00.1360057Z Environment variables 2024-06-26T07:11:00.1360520Z + env 2024-06-26T07:11:00.1362982Z INSTALLED_DB=yes 2024-06-26T07:11:00.1363535Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:00.1364132Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:00.1364699Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:00.1365775Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:00.1366583Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:00.1367296Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:00.1368042Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:00.1368843Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:00.1369432Z HOSTNAME=60a3066c951d 2024-06-26T07:11:00.1370283Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1371361Z GITHUB_ACTION=__self 2024-06-26T07:11:00.1371802Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:00.1375756Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:00.1379552Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:00.1380078Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:00.1380460Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:11:00.1380821Z TEST_CONFIG=inductor_timm 2024-06-26T07:11:00.1381215Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:00.1381704Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:00.1382188Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:00.1382642Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:00.1383086Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:00.1383553Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:00.1384072Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:00.1384557Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:00.1384936Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:00.1385359Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:00.1385777Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1386233Z XLA_CUDA= 2024-06-26T07:11:00.1386746Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:00.1387167Z *** 2024-06-26T07:11:00.1387475Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:00.1387891Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:00.1388278Z GITHUB_ACTIONS=true 2024-06-26T07:11:00.1388633Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:00.1389138Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:00.1389682Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:00.1390173Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1390696Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:00.1391196Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1391997Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:00.1392852Z UCC_HOME=/usr 2024-06-26T07:11:00.1393212Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:00.1393633Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:00.1393998Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:00.1394472Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:00.1395124Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:00.1395606Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:00.1396012Z SHARD_NUMBER=2 2024-06-26T07:11:00.1396350Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:00.1396727Z HOME=/var/lib/jenkins 2024-06-26T07:11:00.1397116Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:00.1397890Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:00.1398391Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:00.1399005Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:11:00.1399405Z CUDA_VERSION=12.1.1 2024-06-26T07:11:00.1399854Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:00.1400328Z NUM_TEST_SHARDS=2 2024-06-26T07:11:00.1400648Z UCX_HOME=/usr 2024-06-26T07:11:00.1401187Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:00.1402261Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1403447Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:00.1404612Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1405718Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:00.1406384Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:00.1406739Z DASHBOARD_TAG= 2024-06-26T07:11:00.1407064Z GITHUB_RUN_ID=9673646808 2024-06-26T07:11:00.1407543Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:00.1408096Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:00.1409095Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1410003Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:00.1410434Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:00.1410837Z PR_NUMBER= 2024-06-26T07:11:00.1411148Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:00.1411484Z VALGRIND=ON 2024-06-26T07:11:00.1411811Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:00.1412277Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:00.1412752Z TERM=xterm 2024-06-26T07:11:00.1413111Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:00.1413548Z INSTALLED_VISION=yes 2024-06-26T07:11:00.1413887Z BRANCH= 2024-06-26T07:11:00.1414191Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:00.1414621Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:00.1415106Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:00.1415887Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:00.1416647Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:00.1417158Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:00.1417649Z REENABLED_ISSUES= 2024-06-26T07:11:00.1417967Z SHLVL=1 2024-06-26T07:11:00.1418254Z MAX_JOBS=14 2024-06-26T07:11:00.1418590Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:00.1418966Z NVARCH=x86_64 2024-06-26T07:11:00.1419285Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:00.1419759Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:00.1432470Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:00.1433060Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:00.1433741Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:00.1434330Z GITHUB_JOB=test 2024-06-26T07:11:00.1434826Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:00.1435401Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:00.1436053Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:00.1436408Z TD_DISTRIBUTED=False 2024-06-26T07:11:00.1436809Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:00.1437266Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:00.1437970Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:00.1438363Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:00.1438730Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:00.1439109Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:00.1440278Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:00.1441481Z GITHUB_BASE_REF= 2024-06-26T07:11:00.1441819Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:00.1442217Z CI=true 2024-06-26T07:11:00.1442548Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:00.1443097Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:00.1443497Z JOB_ID=26688911259 2024-06-26T07:11:00.1443835Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:00.1444200Z GITHUB_HEAD_REF= 2024-06-26T07:11:00.1444521Z GITHUB_ACTION_REF= 2024-06-26T07:11:00.1444972Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:00.1445470Z GITHUB_WORKFLOW=inductor 2024-06-26T07:11:00.1445848Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:00.1446758Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_b2868a4d-2fe9-4e32-a21f-7333594949b0 2024-06-26T07:11:00.1447578Z NO_TD=False 2024-06-26T07:11:00.1447907Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:00.1448288Z _=/usr/bin/env 2024-06-26T07:11:00.1448648Z + echo 'Testing pytorch' 2024-06-26T07:11:00.1449015Z Testing pytorch 2024-06-26T07:11:00.1449357Z + export LANG=C.UTF-8 2024-06-26T07:11:00.1449708Z + LANG=C.UTF-8 2024-06-26T07:11:00.1450040Z + PR_NUMBER= 2024-06-26T07:11:00.1450391Z + [[ inductor_timm == \d\e\f\a\u\l\t ]] 2024-06-26T07:11:00.1450868Z + [[ inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:11:00.1451337Z + [[ inductor_timm == \s\l\o\w ]] 2024-06-26T07:11:00.1451954Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:11:00.1452651Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:11:00.1453213Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:00.1453698Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:00.1454147Z + [[ inductor_timm == *crossref* ]] 2024-06-26T07:11:00.1454707Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:00.1455398Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:11:00.1456055Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:00.1456625Z + pip_install --user ninja==1.10.2 2024-06-26T07:11:00.1457168Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:11:00.4981701Z Collecting ninja==1.10.2 2024-06-26T07:11:00.5159006Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:11:00.5292819Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:11:01.8422403Z Installing collected packages: ninja 2024-06-26T07:11:01.8489507Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:11:01.8490850Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:11:01.8530761Z Successfully installed ninja-1.10.2 2024-06-26T07:11:01.9414659Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:01.9417024Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:01.9418661Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:01.9419581Z + install_tlparse 2024-06-26T07:11:01.9420039Z + pip_install --user tlparse==0.3.7 2024-06-26T07:11:01.9420633Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:11:02.2948435Z Collecting tlparse==0.3.7 2024-06-26T07:11:02.3104189Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:11:02.3206788Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:11:03.6438266Z Installing collected packages: tlparse 2024-06-26T07:11:03.6796573Z Successfully installed tlparse-0.3.7 2024-06-26T07:11:03.7643745Z ++ python -m site --user-base 2024-06-26T07:11:03.7796594Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:03.7798756Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:11:03.7799552Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:11:03.7800222Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:03.7801246Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:11:03.7802400Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:11:03.7803115Z + cd test 2024-06-26T07:11:03.7803652Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:11:05.2645714Z + [[ inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:11:05.2646318Z + [[ inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:11:05.2647161Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:05.2647861Z + pushd test 2024-06-26T07:11:05.2648287Z ~/workspace/test ~/workspace 2024-06-26T07:11:05.2648913Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:11:06.7434012Z + CUDA_VERSION=12.1 2024-06-26T07:11:06.7434592Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:11:06.7435083Z + ISCUDA124= 2024-06-26T07:11:06.7435806Z + popd 2024-06-26T07:11:06.7436196Z ~/workspace 2024-06-26T07:11:06.7439582Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:11:06.7440209Z + [[ inductor_timm == *dynamo_eager* ]] 2024-06-26T07:11:06.7440873Z + [[ inductor_timm == *aot_eager* ]] 2024-06-26T07:11:06.7441525Z + [[ inductor_timm == *aot_inductor* ]] 2024-06-26T07:11:06.7442180Z + [[ inductor_timm == *inductor* ]] 2024-06-26T07:11:06.7442811Z + [[ inductor_timm != *perf* ]] 2024-06-26T07:11:06.7443509Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:11:06.7444145Z + [[ inductor_timm == *dynamic* ]] 2024-06-26T07:11:06.7444764Z + [[ inductor_timm == *cpu_inductor* ]] 2024-06-26T07:11:06.7445527Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:11:06.7469718Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:06.7470428Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:11:06.7472758Z + cd test 2024-06-26T07:11:06.7473456Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:11:08.0906884Z PyTorch built with: 2024-06-26T07:11:08.0907449Z - GCC 9.4 2024-06-26T07:11:08.0907931Z - C++ Version: 201703 2024-06-26T07:11:08.0908826Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:08.0909910Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:08.0910598Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:08.0911139Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:11:08.0911641Z - NNPACK is enabled 2024-06-26T07:11:08.0912040Z - CPU capability usage: AVX2 2024-06-26T07:11:08.0912473Z - CUDA Runtime 12.1 2024-06-26T07:11:08.0913036Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:11:08.0913656Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:11:08.0914416Z - Magma 2.6.1 2024-06-26T07:11:08.0921640Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:11:08.0928168Z 2024-06-26T07:11:08.3674174Z + cd test 2024-06-26T07:11:08.3674969Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:11:09.5870983Z ATen/Parallel: 2024-06-26T07:11:09.5871469Z at::get_num_threads() : 8 2024-06-26T07:11:09.5871893Z at::get_num_interop_threads() : 16 2024-06-26T07:11:09.5872343Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:09.5872767Z omp_get_max_threads() : 8 2024-06-26T07:11:09.5873759Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:09.5874580Z mkl_get_max_threads() : 8 2024-06-26T07:11:09.5875320Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:09.5875971Z std::thread::hardware_concurrency() : 16 2024-06-26T07:11:09.5876425Z Environment variables: 2024-06-26T07:11:09.5876799Z OMP_NUM_THREADS : [not set] 2024-06-26T07:11:09.5877190Z MKL_NUM_THREADS : [not set] 2024-06-26T07:11:09.5877909Z ATen parallel backend: OpenMP 2024-06-26T07:11:09.5878176Z 2024-06-26T07:11:09.8295930Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:09.8296726Z + [[ inductor_timm == *backward* ]] 2024-06-26T07:11:09.8297163Z + [[ inductor_timm == *xla* ]] 2024-06-26T07:11:09.8297587Z + [[ inductor_timm == *executorch* ]] 2024-06-26T07:11:09.8298061Z + [[ inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:11:09.8298720Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:09.8299473Z + [[ inductor_timm == distributed ]] 2024-06-26T07:11:09.8299962Z + [[ inductor_timm == *inductor_distributed* ]] 2024-06-26T07:11:09.8300516Z + [[ inductor_timm == *inductor-halide* ]] 2024-06-26T07:11:09.8301083Z + [[ inductor_timm == *inductor-micro-benchmark* ]] 2024-06-26T07:11:09.8301591Z + [[ inductor_timm == *huggingface* ]] 2024-06-26T07:11:09.8302036Z + [[ inductor_timm == *timm* ]] 2024-06-26T07:11:09.8302442Z + install_torchvision 2024-06-26T07:11:09.8302794Z + local orig_preload 2024-06-26T07:11:09.8303138Z + local commit 2024-06-26T07:11:09.8303481Z ++ get_pinned_commit vision 2024-06-26T07:11:09.8303894Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:11:09.8310829Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:09.8311476Z + orig_preload= 2024-06-26T07:11:09.8311948Z + '[' -n '' ']' 2024-06-26T07:11:09.8312798Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:09.8314220Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:10.1419132Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:10.1423151Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-z9c_3bij 2024-06-26T07:11:10.1440173Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-z9c_3bij 2024-06-26T07:11:11.4664344Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:11:11.4678731Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:12.6824430Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:12.9113776Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:15.2374463Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:11:15.2421510Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:11:15.2424825Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:11:15.2429734Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:11:15.2632303Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:11:15.2638747Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:11:15.2641176Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:11:15.2644341Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:11:15.2646544Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:11:15.2649496Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:11:15.3077129Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:11:15.3230764Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:11:15.3303846Z Building wheels for collected packages: torchvision 2024-06-26T07:12:28.3080063Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:12:28.3107314Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112820 sha256=b40833066496b098fe0ff05199857c868ab85038ae60ddd45a7a3c412b8f8ffa 2024-06-26T07:12:28.3109824Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:12:28.3144558Z Successfully built torchvision 2024-06-26T07:12:29.5191748Z Installing collected packages: torchvision 2024-06-26T07:12:29.8964867Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:12:30.0059233Z + '[' -n '' ']' 2024-06-26T07:12:30.0059567Z + id=1 2024-06-26T07:12:30.0059900Z + test_dynamo_benchmark timm_models 1 2024-06-26T07:12:30.0060742Z ++ pwd 2024-06-26T07:12:30.0063526Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:12:30.0065482Z + local suite=timm_models 2024-06-26T07:12:30.0066124Z + shift 2024-06-26T07:12:30.0066656Z + local shard_id=1 2024-06-26T07:12:30.0067037Z + shift 2024-06-26T07:12:30.0067369Z + [[ inductor_timm == *perf_compare* ]] 2024-06-26T07:12:30.0067829Z + [[ inductor_timm == *perf* ]] 2024-06-26T07:12:30.0068264Z + [[ inductor_timm == *cpu_inductor* ]] 2024-06-26T07:12:30.0069812Z + [[ inductor_timm == *aot_inductor* ]] 2024-06-26T07:12:30.0072568Z + test_single_dynamo_benchmark inference timm_models 1 --inference --bfloat16 2024-06-26T07:12:30.0073492Z ++ pwd 2024-06-26T07:12:30.0074193Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:12:30.0075264Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:12:30.0082111Z + local name=inference 2024-06-26T07:12:30.0082630Z + shift 2024-06-26T07:12:30.0082946Z + local suite=timm_models 2024-06-26T07:12:30.0083384Z + shift 2024-06-26T07:12:30.0083799Z + local shard_id=1 2024-06-26T07:12:30.0084144Z + shift 2024-06-26T07:12:30.0084543Z + partition_flags=() 2024-06-26T07:12:30.0085101Z + local partition_flags 2024-06-26T07:12:30.0085647Z + [[ -n 2 ]] 2024-06-26T07:12:30.0086021Z + [[ -n 1 ]] 2024-06-26T07:12:30.0086626Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:12:30.0087308Z + [[ inductor_timm == *perf_compare* ]] 2024-06-26T07:12:30.0087773Z + [[ inductor_timm == *perf* ]] 2024-06-26T07:12:30.0088201Z + [[ inductor_timm == *aot_inductor* ]] 2024-06-26T07:12:30.0089762Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-06-26T07:12:35.8035972Z 2024-06-26T07:12:35.9618910Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:12:35.9619538Z 2024-06-26T07:12:36.0208976Z model.safetensors: 0% 0.00/29.6M [00:00 will be ignored 2024-06-26T07:32:27.0832297Z pass 2024-06-26T07:32:27.0876614Z TIMING: entire_frame_compile:69.4281 code_gen:29.84966 inductor_compile:50.20942 backend_compile:57.97214 2024-06-26T07:32:27.0878148Z STATS: call_* op count: 797 | FakeTensor.__torch_dispatch__:16656 | FakeTensorMode.__torch_dispatch__:84942 | ProxyTorchDispatchMode.__torch_dispatch__:17697 2024-06-26T07:32:27.0879368Z Dynamo produced 4 graphs covering 797 ops with 7 graph breaks (5 unique) 2024-06-26T07:32:34.1963785Z 2024-06-26T07:32:34.9093962Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:34.9094467Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:34.9097181Z cuda train mobilenetv2_100 2024-06-26T07:33:15.9949840Z W0626 07:33:15.994000 140678909112960 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:33:59.9340924Z pass 2024-06-26T07:33:59.9387151Z TIMING: entire_frame_compile:66.77087 code_gen:27.15331 inductor_compile:49.27693 backend_compile:55.37083 2024-06-26T07:33:59.9389715Z STATS: call_* op count: 796 | FakeTensor.__torch_dispatch__:17634 | FakeTensorMode.__torch_dispatch__:86783 | ProxyTorchDispatchMode.__torch_dispatch__:17713 2024-06-26T07:33:59.9390905Z Dynamo produced 3 graphs covering 796 ops with 7 graph breaks (5 unique) 2024-06-26T07:34:07.0187961Z 2024-06-26T07:34:07.8027949Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:07.8028447Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:07.8028905Z cuda train mobilenetv3_large_100 2024-06-26T07:34:52.5576264Z W0626 07:34:52.556000 139641806148224 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:35:42.4526389Z pass 2024-06-26T07:35:42.4595356Z TIMING: entire_frame_compile:74.00124 code_gen:32.2933 inductor_compile:56.1438 backend_compile:61.75773 2024-06-26T07:35:42.4598594Z STATS: call_* op count: 855 | FakeTensor.__torch_dispatch__:19057 | FakeTensorMode.__torch_dispatch__:93236 | ProxyTorchDispatchMode.__torch_dispatch__:19569 2024-06-26T07:35:42.4599754Z Dynamo produced 3 graphs covering 855 ops with 7 graph breaks (5 unique) 2024-06-26T07:35:49.8516999Z 2024-06-26T07:35:50.6262563Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:50.6263496Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:50.6264408Z cuda train mobilevit_s 2024-06-26T07:37:08.6510450Z pass 2024-06-26T07:37:08.6587926Z TIMING: entire_frame_compile:34.42037 code_gen:40.46704 inductor_compile:57.80145 backend_compile:29.79131 2024-06-26T07:37:08.6589541Z STATS: call_* op count: 503 | FakeTensor.__torch_dispatch__:11221 | FakeTensorMode.__torch_dispatch__:58485 | ProxyTorchDispatchMode.__torch_dispatch__:16472 2024-06-26T07:37:08.6590694Z Dynamo produced 3 graphs covering 503 ops with 6 graph breaks (5 unique) 2024-06-26T07:37:14.3401727Z 2024-06-26T07:37:15.4278297Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:15.4278903Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:37:15.4279455Z cuda train nfnet_l0 2024-06-26T07:38:00.0507273Z W0626 07:38:00.049000 139864525529728 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:39:05.9627665Z pass 2024-06-26T07:39:05.9856894Z TIMING: entire_frame_compile:87.55256 code_gen:31.43978 inductor_compile:56.93978 backend_compile:68.26762 2024-06-26T07:39:05.9859922Z STATS: call_* op count: 1232 | FakeTensorMode.__torch_dispatch__:113308 | FakeTensor.__torch_dispatch__:21204 | ProxyTorchDispatchMode.__torch_dispatch__:25953 2024-06-26T07:39:05.9861122Z Dynamo produced 3 graphs covering 1232 ops with 7 graph breaks (5 unique) 2024-06-26T07:39:13.7165472Z 2024-06-26T07:39:14.6649898Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:14.6650414Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:14.6651063Z cuda train pit_b_224 2024-06-26T07:39:47.0604164Z pass 2024-06-26T07:39:47.1077031Z TIMING: entire_frame_compile:15.8488 code_gen:10.5153 inductor_compile:18.17579 backend_compile:13.56124 2024-06-26T07:39:47.1078797Z STATS: call_* op count: 404 | FakeTensor.__torch_dispatch__:7224 | FakeTensorMode.__torch_dispatch__:41255 | ProxyTorchDispatchMode.__torch_dispatch__:12561 2024-06-26T07:39:47.1079934Z Dynamo produced 2 graphs covering 404 ops with 6 graph breaks (5 unique) 2024-06-26T07:39:51.7603615Z 2024-06-26T07:39:53.2948151Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:53.2948895Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:53.2949477Z cuda train pnasnet5large 2024-06-26T07:44:18.4421891Z pass 2024-06-26T07:44:18.5562934Z TIMING: entire_frame_compile:109.31139 code_gen:160.45646 inductor_compile:203.41223 backend_compile:98.73769 2024-06-26T07:44:18.5564359Z STATS: call_* op count: 963 | FakeTensor.__torch_dispatch__:38589 | FakeTensorMode.__torch_dispatch__:198131 | ProxyTorchDispatchMode.__torch_dispatch__:49820 2024-06-26T07:44:18.5566769Z Dynamo produced 2 graphs covering 963 ops with 5 graph breaks (4 unique) 2024-06-26T07:44:30.9745821Z 2024-06-26T07:44:32.2449142Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:44:32.2449729Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:44:32.2450201Z cuda train poolformer_m36 2024-06-26T07:48:03.0587261Z pass 2024-06-26T07:48:03.0981200Z TIMING: entire_frame_compile:57.64785 code_gen:156.51258 inductor_compile:180.07463 backend_compile:50.49566 2024-06-26T07:48:03.0982561Z STATS: call_* op count: 741 | FakeTensor.__torch_dispatch__:16757 | FakeTensorMode.__torch_dispatch__:80099 | ProxyTorchDispatchMode.__torch_dispatch__:22827 2024-06-26T07:48:03.0983714Z Dynamo produced 2 graphs covering 741 ops with 6 graph breaks (5 unique) 2024-06-26T07:48:11.0690190Z 2024-06-26T07:48:11.7823646Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:11.7824505Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:11.7824961Z cuda train regnety_002 2024-06-26T07:48:50.7446462Z pass 2024-06-26T07:48:50.7482223Z TIMING: entire_frame_compile:23.84411 code_gen:13.20586 inductor_compile:23.19123 backend_compile:19.30181 2024-06-26T07:48:50.7483543Z STATS: call_* op count: 369 | FakeTensor.__torch_dispatch__:9014 | FakeTensorMode.__torch_dispatch__:45506 | ProxyTorchDispatchMode.__torch_dispatch__:12209 2024-06-26T07:48:50.7484687Z Dynamo produced 2 graphs covering 369 ops with 6 graph breaks (5 unique) 2024-06-26T07:48:55.8326343Z 2024-06-26T07:48:56.7604791Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:56.7605287Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:56.7608589Z cuda train repvgg_a2 2024-06-26T07:49:39.9981049Z W0626 07:49:39.996000 140109170446976 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:50:25.9667612Z pass 2024-06-26T07:50:25.9822745Z TIMING: entire_frame_compile:71.41847 code_gen:27.78364 inductor_compile:49.23673 backend_compile:58.29279 2024-06-26T07:50:25.9825437Z STATS: call_* op count: 920 | FakeTensor.__torch_dispatch__:17660 | FakeTensorMode.__torch_dispatch__:90130 | ProxyTorchDispatchMode.__torch_dispatch__:18975 2024-06-26T07:50:25.9826597Z Dynamo produced 3 graphs covering 920 ops with 7 graph breaks (5 unique) 2024-06-26T07:50:33.2572433Z 2024-06-26T07:50:34.5897352Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:50:34.5897853Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:50:34.5898318Z cuda train res2net101_26w_4s 2024-06-26T07:52:49.8621009Z pass 2024-06-26T07:52:49.8943550Z TIMING: entire_frame_compile:79.45914 code_gen:57.03253 inductor_compile:93.11856 backend_compile:74.04097 2024-06-26T07:52:49.8945492Z STATS: call_* op count: 807 | FakeTensor.__torch_dispatch__:33531 | FakeTensorMode.__torch_dispatch__:158054 | ProxyTorchDispatchMode.__torch_dispatch__:37834 2024-06-26T07:52:49.8946670Z Dynamo produced 2 graphs covering 807 ops with 6 graph breaks (5 unique) 2024-06-26T07:53:00.1464381Z 2024-06-26T07:53:01.2408433Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:01.2408930Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:53:01.2409396Z cuda train res2net50_14w_8s 2024-06-26T07:55:01.0060399Z pass 2024-06-26T07:55:01.0252072Z TIMING: entire_frame_compile:69.22079 code_gen:50.8694 inductor_compile:82.83477 backend_compile:64.65444 2024-06-26T07:55:01.0253381Z STATS: call_* op count: 703 | FakeTensor.__torch_dispatch__:29747 | FakeTensorMode.__torch_dispatch__:143048 | ProxyTorchDispatchMode.__torch_dispatch__:33247 2024-06-26T07:55:01.0259289Z Dynamo produced 2 graphs covering 703 ops with 6 graph breaks (5 unique) 2024-06-26T07:55:10.2897724Z 2024-06-26T07:55:11.2327691Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:11.2328326Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:11.2328794Z cuda train res2next50 2024-06-26T07:56:17.8494535Z pass 2024-06-26T07:56:17.8664390Z TIMING: entire_frame_compile:36.46892 code_gen:26.7063 inductor_compile:44.63322 backend_compile:32.85192 2024-06-26T07:56:17.8668891Z STATS: call_* op count: 399 | FakeTensor.__torch_dispatch__:16673 | FakeTensorMode.__torch_dispatch__:78559 | ProxyTorchDispatchMode.__torch_dispatch__:18879 2024-06-26T07:56:17.8671278Z Dynamo produced 2 graphs covering 399 ops with 6 graph breaks (5 unique) 2024-06-26T07:56:24.0109036Z 2024-06-26T07:56:24.6160177Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:56:24.6160689Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:56:24.6163533Z cuda train resmlp_12_224 2024-06-26T07:56:45.1269788Z pass 2024-06-26T07:56:45.1392750Z TIMING: entire_frame_compile:10.99037 code_gen:6.05401 inductor_compile:10.53434 backend_compile:8.85333 2024-06-26T07:56:45.1394061Z STATS: call_* op count: 216 | FakeTensor.__torch_dispatch__:4403 | FakeTensorMode.__torch_dispatch__:25576 | ProxyTorchDispatchMode.__torch_dispatch__:8345 2024-06-26T07:56:45.1395587Z Dynamo produced 2 graphs covering 216 ops with 6 graph breaks (5 unique) 2024-06-26T07:56:49.4474702Z 2024-06-26T07:56:50.8952011Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:56:50.8952521Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:56:50.8952985Z cuda train resnest101e 2024-06-26T07:58:50.8323238Z pass 2024-06-26T07:58:50.8719874Z TIMING: entire_frame_compile:64.17465 code_gen:44.69022 inductor_compile:78.66006 backend_compile:57.87494 2024-06-26T07:58:50.8723043Z STATS: call_* op count: 1022 | FakeTensor.__torch_dispatch__:29732 | FakeTensorMode.__torch_dispatch__:145831 | ProxyTorchDispatchMode.__torch_dispatch__:38715 2024-06-26T07:58:50.8724990Z Dynamo produced 2 graphs covering 1022 ops with 6 graph breaks (5 unique) 2024-06-26T07:58:59.5250800Z 2024-06-26T07:59:00.4404437Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:59:00.4404945Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:59:00.4405403Z cuda train rexnet_100 2024-06-26T08:00:04.1494657Z W0626 08:00:04.148000 140678851515008 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:01:10.0634397Z pass 2024-06-26T08:01:10.0714507Z TIMING: entire_frame_compile:99.70914 code_gen:47.15815 inductor_compile:77.64064 backend_compile:82.87333 2024-06-26T08:01:10.0716873Z STATS: call_* op count: 1104 | FakeTensor.__torch_dispatch__:25154 | FakeTensorMode.__torch_dispatch__:123572 | ProxyTorchDispatchMode.__torch_dispatch__:26051 2024-06-26T08:01:10.0718213Z Dynamo produced 3 graphs covering 1104 ops with 7 graph breaks (5 unique) 2024-06-26T08:01:18.9035585Z 2024-06-26T08:01:19.8330897Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:19.8331504Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:19.8332024Z cuda train sebotnet33ts_256 2024-06-26T08:02:22.3224551Z pass 2024-06-26T08:02:22.3348029Z TIMING: entire_frame_compile:35.54547 code_gen:27.06271 inductor_compile:40.53728 backend_compile:31.02731 2024-06-26T08:02:22.3350504Z STATS: call_* op count: 466 | FakeTensor.__torch_dispatch__:9493 | FakeTensorMode.__torch_dispatch__:50071 | ProxyTorchDispatchMode.__torch_dispatch__:13430 2024-06-26T08:02:22.3351683Z Dynamo produced 3 graphs covering 466 ops with 6 graph breaks (5 unique) 2024-06-26T08:02:27.9298639Z 2024-06-26T08:02:28.9632241Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:02:28.9633508Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:02:28.9634420Z cuda train selecsls42b 2024-06-26T08:02:59.4207625Z pass 2024-06-26T08:02:59.4473834Z TIMING: entire_frame_compile:17.50303 code_gen:10.90904 inductor_compile:19.30091 backend_compile:15.93386 2024-06-26T08:02:59.4477977Z STATS: call_* op count: 137 | FakeTensor.__torch_dispatch__:7417 | FakeTensorMode.__torch_dispatch__:35237 | ProxyTorchDispatchMode.__torch_dispatch__:8917 2024-06-26T08:02:59.4479138Z Dynamo produced 2 graphs covering 137 ops with 6 graph breaks (5 unique) 2024-06-26T08:03:04.2689007Z 2024-06-26T08:03:04.9813115Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:03:04.9813851Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:03:04.9814457Z cuda train spnasnet_100 2024-06-26T08:03:52.1185414Z W0626 08:03:52.117000 140440334164608 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:04:46.3090163Z pass 2024-06-26T08:04:46.3130623Z TIMING: entire_frame_compile:81.87624 code_gen:31.97196 inductor_compile:57.67695 backend_compile:68.08632 2024-06-26T08:04:46.3131946Z STATS: call_* op count: 976 | FakeTensor.__torch_dispatch__:20401 | FakeTensorMode.__torch_dispatch__:104011 | ProxyTorchDispatchMode.__torch_dispatch__:21710 2024-06-26T08:04:46.3133126Z Dynamo produced 3 graphs covering 976 ops with 7 graph breaks (5 unique) 2024-06-26T08:04:54.1921450Z 2024-06-26T08:04:55.8822575Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:04:55.8823441Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:04:55.8826157Z cuda train swin_base_patch4_window7_224 2024-06-26T08:06:30.0135029Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:06:30.0136201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:06:30.0137176Z pred = mod(*cloned_inputs) 2024-06-26T08:06:30.0141351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:06:30.0142464Z return forward_call(*args, **kwargs) 2024-06-26T08:06:30.0143465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 617, in forward 2024-06-26T08:06:30.0144307Z x = self.forward_features(x) 2024-06-26T08:06:30.0145610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 609, in forward_features 2024-06-26T08:06:30.0146821Z x = self.layers(x) 2024-06-26T08:06:30.0147685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:06:30.0148537Z return forward_call(*args, **kwargs) 2024-06-26T08:06:30.0149454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 443, in forward 2024-06-26T08:06:30.0150283Z x = self.blocks(x) 2024-06-26T08:06:30.0151130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:06:30.0151980Z return forward_call(*args, **kwargs) 2024-06-26T08:06:30.0152900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 323, in forward 2024-06-26T08:06:30.0153804Z x = x + self.drop_path1(self._attn(self.norm1(x))) 2024-06-26T08:06:30.0154932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 307, in _attn 2024-06-26T08:06:30.0156019Z attn_windows = self.attn(x_windows, mask=self.attn_mask) # nW*B, window_size*window_size, C 2024-06-26T08:06:30.0157199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:06:30.0158428Z return forward_call(*args, **kwargs) 2024-06-26T08:06:30.0159362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 172, in forward 2024-06-26T08:06:30.0160213Z attn = attn + self._get_rel_pos_bias() 2024-06-26T08:06:30.0161206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 143, in _get_rel_pos_bias 2024-06-26T08:06:30.0162199Z relative_position_bias = self.relative_position_bias_table[ 2024-06-26T08:06:30.0162610Z 2024-06-26T08:06:30.5814128Z W0626 08:06:30.580000 140557922644608 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:07:55.9137303Z pass 2024-06-26T08:07:55.9582018Z TIMING: entire_frame_compile:134.49798 code_gen:61.18342 inductor_compile:102.78446 backend_compile:115.06181 2024-06-26T08:07:55.9584296Z STATS: call_* op count: 2473 | FakeTensor.__torch_dispatch__:31970 | FakeTensorMode.__torch_dispatch__:201026 | ProxyTorchDispatchMode.__torch_dispatch__:50112 2024-06-26T08:07:55.9585457Z Dynamo produced 3 graphs covering 2473 ops with 7 graph breaks (5 unique) 2024-06-26T08:08:05.9655988Z 2024-06-26T08:08:08.8618319Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:08:08.8618923Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:08:08.8623015Z cuda train swsl_resnext101_32x16d 2024-06-26T08:09:20.0939767Z pass 2024-06-26T08:09:20.2277674Z TIMING: entire_frame_compile:37.83547 code_gen:19.9918 inductor_compile:38.47272 backend_compile:34.52203 2024-06-26T08:09:20.2279517Z STATS: call_* op count: 415 | FakeTensor.__torch_dispatch__:18037 | FakeTensorMode.__torch_dispatch__:85923 | ProxyTorchDispatchMode.__torch_dispatch__:22001 2024-06-26T08:09:20.2281002Z Dynamo produced 2 graphs covering 415 ops with 6 graph breaks (5 unique) 2024-06-26T08:09:26.5269987Z 2024-06-26T08:09:27.3109468Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:09:27.3110387Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:09:27.3111285Z cuda train tf_efficientnet_b0 2024-06-26T08:10:20.6077934Z pass 2024-06-26T08:10:20.6159374Z TIMING: entire_frame_compile:31.48195 code_gen:20.8844 inductor_compile:34.91691 backend_compile:26.56947 2024-06-26T08:10:20.6161346Z STATS: call_* op count: 371 | FakeTensorMode.__torch_dispatch__:55697 | FakeTensor.__torch_dispatch__:11937 | ProxyTorchDispatchMode.__torch_dispatch__:14649 2024-06-26T08:10:20.6162506Z Dynamo produced 2 graphs covering 371 ops with 6 graph breaks (5 unique) 2024-06-26T08:10:26.5219481Z 2024-06-26T08:10:27.4187507Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:27.4188091Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:27.4188557Z cuda train tf_mixnet_l 2024-06-26T08:11:42.0247570Z pass 2024-06-26T08:11:42.0334070Z TIMING: entire_frame_compile:44.41681 code_gen:30.61362 inductor_compile:49.04141 backend_compile:37.50073 2024-06-26T08:11:42.0337477Z STATS: call_* op count: 692 | FakeTensorMode.__torch_dispatch__:79852 | FakeTensor.__torch_dispatch__:15299 | ProxyTorchDispatchMode.__torch_dispatch__:19823 2024-06-26T08:11:42.0338668Z Dynamo produced 2 graphs covering 692 ops with 6 graph breaks (5 unique) 2024-06-26T08:11:48.9744292Z 2024-06-26T08:11:49.8018987Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:49.8019920Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:49.8020809Z cuda train tinynet_a 2024-06-26T08:12:48.9903590Z pass 2024-06-26T08:12:48.9982759Z TIMING: entire_frame_compile:35.3253 code_gen:22.22094 inductor_compile:38.51396 backend_compile:29.91369 2024-06-26T08:12:48.9985593Z STATS: call_* op count: 435 | FakeTensor.__torch_dispatch__:14095 | FakeTensorMode.__torch_dispatch__:65530 | ProxyTorchDispatchMode.__torch_dispatch__:17283 2024-06-26T08:12:48.9986797Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-06-26T08:12:55.1820764Z 2024-06-26T08:12:56.1362747Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:56.1363258Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:56.1363720Z cuda train tnt_s_patch16_224 2024-06-26T08:14:05.2470408Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:14:05.2472265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:14:05.2473336Z pred = mod(*cloned_inputs) 2024-06-26T08:14:05.2474414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:14:05.2475513Z return forward_call(*args, **kwargs) 2024-06-26T08:14:05.2476464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 333, in forward 2024-06-26T08:14:05.2477663Z x = self.forward_features(x) 2024-06-26T08:14:05.2486951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 309, in forward_features 2024-06-26T08:14:05.2487879Z pixel_embed = self.pixel_embed(x, self.pixel_pos) 2024-06-26T08:14:05.2488968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:14:05.2489951Z return forward_call(*args, **kwargs) 2024-06-26T08:14:05.2490813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 182, in forward 2024-06-26T08:14:05.2491568Z x = self.unfold(x) 2024-06-26T08:14:05.2491791Z 2024-06-26T08:14:05.4469619Z W0626 08:14:05.446000 140070578729600 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:15:31.8628187Z pass 2024-06-26T08:15:31.8867956Z TIMING: entire_frame_compile:122.13925 code_gen:47.24667 inductor_compile:87.08104 backend_compile:105.49935 2024-06-26T08:15:31.8869624Z STATS: call_* op count: 1922 | FakeTensor.__torch_dispatch__:32685 | FakeTensorMode.__torch_dispatch__:196624 | ProxyTorchDispatchMode.__torch_dispatch__:49260 2024-06-26T08:15:31.8870989Z Dynamo produced 3 graphs covering 1922 ops with 7 graph breaks (5 unique) 2024-06-26T08:15:42.4466834Z 2024-06-26T08:15:43.6578674Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:43.6579179Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:15:43.6579639Z cuda train twins_pcpvt_base 2024-06-26T08:17:00.8384582Z W0626 08:17:00.837000 139722966196864 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:19:19.0853838Z pass 2024-06-26T08:19:19.1153239Z TIMING: entire_frame_compile:168.20855 code_gen:65.66791 inductor_compile:121.83049 backend_compile:142.65534 2024-06-26T08:19:19.1156282Z STATS: call_* op count: 2517 | FakeTensor.__torch_dispatch__:47362 | FakeTensorMode.__torch_dispatch__:258286 | ProxyTorchDispatchMode.__torch_dispatch__:59071 2024-06-26T08:19:19.1157849Z Dynamo produced 3 graphs covering 2517 ops with 7 graph breaks (5 unique) 2024-06-26T08:19:31.6953562Z 2024-06-26T08:19:32.9175998Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:19:32.9176485Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:19:32.9177769Z cuda train visformer_small 2024-06-26T08:20:07.0297290Z W0626 08:20:07.028000 140357089030784 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:20:42.5574915Z pass 2024-06-26T08:20:42.5820401Z TIMING: entire_frame_compile:54.53202 code_gen:20.53816 inductor_compile:36.55077 backend_compile:46.13617 2024-06-26T08:20:42.5822789Z STATS: call_* op count: 699 | FakeTensor.__torch_dispatch__:13870 | FakeTensorMode.__torch_dispatch__:72850 | ProxyTorchDispatchMode.__torch_dispatch__:15868 2024-06-26T08:20:42.5823965Z Dynamo produced 3 graphs covering 699 ops with 7 graph breaks (5 unique) 2024-06-26T08:20:48.5926451Z 2024-06-26T08:20:50.0223093Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:20:50.0223595Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:20:50.0227341Z cuda train vit_base_patch16_224 2024-06-26T08:21:16.6928168Z W0626 08:21:16.691000 140269492695680 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:21:55.9323034Z pass 2024-06-26T08:21:55.9813107Z TIMING: entire_frame_compile:49.58133 code_gen:19.39759 inductor_compile:35.15519 backend_compile:42.18592 2024-06-26T08:21:55.9814484Z STATS: call_* op count: 829 | FakeTensor.__torch_dispatch__:13986 | FakeTensorMode.__torch_dispatch__:75674 | ProxyTorchDispatchMode.__torch_dispatch__:17807 2024-06-26T08:21:55.9818926Z Dynamo produced 3 graphs covering 829 ops with 7 graph breaks (5 unique) 2024-06-26T08:22:02.0401284Z 2024-06-26T08:22:03.0275482Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:22:03.0276095Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:22:03.0276556Z cuda train volo_d1_224 2024-06-26T08:22:40.8871310Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:22:40.8873899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:22:40.8875305Z pred = mod(*cloned_inputs) 2024-06-26T08:22:40.8876331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:40.8877186Z return forward_call(*args, **kwargs) 2024-06-26T08:22:40.8878269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 725, in forward 2024-06-26T08:22:40.8881832Z x = self.forward_features(x) 2024-06-26T08:22:40.8883057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 698, in forward_features 2024-06-26T08:22:40.8884264Z x = self.forward_tokens(x) 2024-06-26T08:22:40.8885339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 625, in forward_tokens 2024-06-26T08:22:40.8886402Z x = block(x) 2024-06-26T08:22:40.8887227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:40.8888075Z return forward_call(*args, **kwargs) 2024-06-26T08:22:40.8888922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 134, in forward 2024-06-26T08:22:40.8889759Z x = x + self.drop_path(self.attn(self.norm1(x))) 2024-06-26T08:22:40.8890724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:22:40.8891729Z return forward_call(*args, **kwargs) 2024-06-26T08:22:40.8892589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 88, in forward 2024-06-26T08:22:40.8893660Z x = F.fold(x, output_size=(H, W), kernel_size=self.kernel_size, padding=self.padding, stride=self.stride) 2024-06-26T08:22:40.8894257Z 2024-06-26T08:23:08.1571323Z W0626 08:23:08.156000 140444702503552 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:24:12.1070395Z pass 2024-06-26T08:24:12.1186746Z TIMING: entire_frame_compile:95.75019 code_gen:46.07799 inductor_compile:75.92277 backend_compile:83.33342 2024-06-26T08:24:12.1188500Z STATS: call_* op count: 1410 | FakeTensor.__torch_dispatch__:23496 | FakeTensorMode.__torch_dispatch__:143231 | ProxyTorchDispatchMode.__torch_dispatch__:35802 2024-06-26T08:24:12.1189687Z Dynamo produced 3 graphs covering 1410 ops with 7 graph breaks (5 unique) 2024-06-26T08:24:20.5030070Z 2024-06-26T08:24:23.6508385Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:24:23.6509318Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:24:23.6510244Z cuda train xcit_large_24_p8_224 2024-06-26T08:24:23.7404934Z WARNING:common:fp64 golden ref were not generated for xcit_large_24_p8_224. Setting accuracy check to cosine 2024-06-26T08:26:12.8853374Z W0626 08:26:12.884000 139881272042112 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:28:58.6286770Z pass_due_to_skip 2024-06-26T08:28:58.6705314Z TIMING: entire_frame_compile:213.49466 code_gen:83.52015 inductor_compile:155.52289 backend_compile:181.39463 2024-06-26T08:28:58.6707143Z STATS: call_* op count: 3114 | FakeTensor.__torch_dispatch__:61154 | FakeTensorMode.__torch_dispatch__:326702 | ProxyTorchDispatchMode.__torch_dispatch__:78435 2024-06-26T08:28:58.6708358Z Dynamo produced 3 graphs covering 3114 ops with 7 graph breaks (5 unique) 2024-06-26T08:29:11.2931567Z accuracy pass_rate=96.77% 2024-06-26T08:29:11.2935550Z calls_captured gmean=744.81x mean=934.323x 2024-06-26T08:29:11.2939213Z unique_graphs gmean=2.49x mean=2.548x 2024-06-26T08:29:11.2943168Z graph_breaks gmean=6.39x mean=6.419x 2024-06-26T08:29:11.2946978Z unique_graph_breaks gmean=4.96x mean=4.968x 2024-06-26T08:29:11.2950778Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:29:11.2954583Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:29:11.2958770Z cudagraph_skips gmean=0.00x mean=0.097x 2024-06-26T08:29:13.0901522Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_timm_training.csv 2024-06-26T08:29:13.3408887Z mixnet_l PASS 2024-06-26T08:29:13.3418174Z mnasnet_100 PASS 2024-06-26T08:29:13.3421579Z mobilenetv2_100 PASS 2024-06-26T08:29:13.3426921Z mobilenetv3_large_100 PASS 2024-06-26T08:29:13.3432041Z mobilevit_s PASS 2024-06-26T08:29:13.3436974Z nfnet_l0 PASS 2024-06-26T08:29:13.3442471Z pit_b_224 PASS 2024-06-26T08:29:13.3447396Z pnasnet5large PASS 2024-06-26T08:29:13.3452122Z poolformer_m36 PASS 2024-06-26T08:29:13.3457114Z regnety_002 PASS 2024-06-26T08:29:13.3461855Z repvgg_a2 PASS 2024-06-26T08:29:13.3466789Z res2net101_26w_4s PASS 2024-06-26T08:29:13.3471490Z res2net50_14w_8s PASS 2024-06-26T08:29:13.3476548Z res2next50 PASS 2024-06-26T08:29:13.3481806Z resmlp_12_224 PASS 2024-06-26T08:29:13.3486718Z resnest101e PASS 2024-06-26T08:29:13.3491595Z rexnet_100 PASS 2024-06-26T08:29:13.3498454Z sebotnet33ts_256 PASS 2024-06-26T08:29:13.3503001Z selecsls42b PASS 2024-06-26T08:29:13.3507825Z spnasnet_100 PASS 2024-06-26T08:29:13.3512780Z swin_base_patch4_window7_224 PASS 2024-06-26T08:29:13.3518424Z swsl_resnext101_32x16d PASS 2024-06-26T08:29:13.3523163Z tf_efficientnet_b0 PASS 2024-06-26T08:29:13.3527909Z tf_mixnet_l PASS 2024-06-26T08:29:13.3532544Z tinynet_a PASS 2024-06-26T08:29:13.3537379Z tnt_s_patch16_224 PASS 2024-06-26T08:29:13.3542063Z twins_pcpvt_base PASS 2024-06-26T08:29:13.3546979Z visformer_small PASS 2024-06-26T08:29:13.3551574Z vit_base_patch16_224 PASS 2024-06-26T08:29:13.3556718Z volo_d1_224 PASS 2024-06-26T08:29:13.3561633Z xcit_large_24_p8_224 XFAIL 2024-06-26T08:29:13.3943508Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_timm_training.csv 2024-06-26T08:29:13.6659590Z mixnet_l PASS 2024-06-26T08:29:13.6663296Z mnasnet_100 PASS 2024-06-26T08:29:13.6668787Z mobilenetv2_100 PASS 2024-06-26T08:29:13.6673593Z mobilenetv3_large_100 PASS 2024-06-26T08:29:13.6679244Z mobilevit_s PASS 2024-06-26T08:29:13.6683959Z nfnet_l0 PASS 2024-06-26T08:29:13.6689081Z pit_b_224 PASS 2024-06-26T08:29:13.6694829Z pnasnet5large PASS 2024-06-26T08:29:13.6700053Z poolformer_m36 PASS 2024-06-26T08:29:13.6705836Z regnety_002 PASS 2024-06-26T08:29:13.6710966Z repvgg_a2 PASS 2024-06-26T08:29:13.6716650Z res2net101_26w_4s PASS 2024-06-26T08:29:13.6722812Z res2net50_14w_8s PASS 2024-06-26T08:29:13.6728106Z res2next50 PASS 2024-06-26T08:29:13.6733855Z resmlp_12_224 PASS 2024-06-26T08:29:13.6739335Z resnest101e PASS 2024-06-26T08:29:13.6745036Z rexnet_100 PASS 2024-06-26T08:29:13.6750454Z sebotnet33ts_256 PASS 2024-06-26T08:29:13.6756293Z selecsls42b PASS 2024-06-26T08:29:13.6762271Z spnasnet_100 PASS 2024-06-26T08:29:13.6767812Z swin_base_patch4_window7_224 PASS 2024-06-26T08:29:13.6773293Z swsl_resnext101_32x16d PASS 2024-06-26T08:29:13.6778927Z tf_efficientnet_b0 PASS 2024-06-26T08:29:13.6784456Z tf_mixnet_l PASS 2024-06-26T08:29:13.6790028Z tinynet_a PASS 2024-06-26T08:29:13.6795603Z tnt_s_patch16_224 PASS 2024-06-26T08:29:13.6801543Z twins_pcpvt_base PASS 2024-06-26T08:29:13.6806897Z visformer_small PASS 2024-06-26T08:29:13.6812536Z vit_base_patch16_224 PASS 2024-06-26T08:29:13.6817915Z volo_d1_224 PASS 2024-06-26T08:29:13.6823594Z xcit_large_24_p8_224 PASS 2024-06-26T08:29:13.7217380Z + cleanup_workspace 2024-06-26T08:29:13.7218659Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:29:13.7219851Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:29:13.7220931Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:29:13.7221826Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:29:13.7223002Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:29:13.7224205Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:29:13.7225344Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:29:14.1913842Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:29:14.1914390Z cat test/**/*_toprint.log || true 2024-06-26T08:29:14.1922505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:14.1923017Z env: 2024-06-26T08:29:14.1923311Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.1923780Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.1924518Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.1925166Z ##[endgroup] 2024-06-26T08:29:14.1988568Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:29:14.2016985Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:29:14.2017473Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:29:14.2025107Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:14.2025618Z env: 2024-06-26T08:29:14.2025923Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.2026404Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.2027145Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.2027823Z MONITOR_SCRIPT_PID: 11266 2024-06-26T08:29:14.2028204Z ##[endgroup] 2024-06-26T08:29:14.2212312Z Prepare all required actions 2024-06-26T08:29:14.2212781Z Getting action download info 2024-06-26T08:29:14.4090530Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:29:14.5767800Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:29:14.5768300Z with: 2024-06-26T08:29:14.5768818Z file-suffix: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259 2024-06-26T08:29:14.5769465Z s3-bucket: gha-artifacts 2024-06-26T08:29:14.5769824Z env: 2024-06-26T08:29:14.5770127Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.5770601Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.5771351Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.5771999Z ##[endgroup] 2024-06-26T08:29:14.5797892Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:29:14.5798553Z # Remove any previous test jsons if they exist 2024-06-26T08:29:14.5799077Z rm -f test-jsons-*.zip 2024-06-26T08:29:14.5799632Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:29:14.5807647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:14.5808166Z env: 2024-06-26T08:29:14.5808466Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.5808942Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.5809692Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.5810562Z FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259 2024-06-26T08:29:14.5811179Z ##[endgroup] 2024-06-26T08:29:14.5978452Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:29:14.6007508Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:29:14.6008385Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:29:14.6014166Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:29:14.6040604Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:29:14.6041267Z # Remove any previous test reports if they exist 2024-06-26T08:29:14.6041807Z rm -f test-reports-*.zip 2024-06-26T08:29:14.6042414Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:29:14.6050131Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:14.6050646Z env: 2024-06-26T08:29:14.6050947Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.6051422Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.6052214Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.6053087Z FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259 2024-06-26T08:29:14.6053722Z ##[endgroup] 2024-06-26T08:29:14.6208878Z adding: test/test-reports/inference_timm_models.csv (deflated 68%) 2024-06-26T08:29:14.6209684Z adding: test/test-reports/training_timm_models.csv (deflated 66%) 2024-06-26T08:29:14.6213445Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-06-26T08:29:14.6214393Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 74%) 2024-06-26T08:29:14.6239954Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:29:14.6240593Z # Remove any previous usage logs if they exist 2024-06-26T08:29:14.6241110Z rm -f logs-*.zip 2024-06-26T08:29:14.6241791Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:29:14.6242642Z # so check to see if the file exists first 2024-06-26T08:29:14.6243156Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:29:14.6243714Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:29:14.6244215Z fi 2024-06-26T08:29:14.6244715Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:29:14.6245305Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:29:14.6245814Z fi 2024-06-26T08:29:14.6253244Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:14.6253763Z env: 2024-06-26T08:29:14.6254061Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.6254533Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.6255278Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.6256142Z FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259 2024-06-26T08:29:14.6256762Z ##[endgroup] 2024-06-26T08:29:14.6327388Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:29:14.6391737Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:29:14.6392475Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:29:14.6393043Z rm -f debug-*.zip 2024-06-26T08:29:14.6393448Z if [ -d 'test/debug' ]; then 2024-06-26T08:29:14.6393981Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:29:14.6394475Z fi 2024-06-26T08:29:14.6402158Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:14.6402698Z env: 2024-06-26T08:29:14.6403003Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.6403472Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.6404208Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.6405079Z FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259 2024-06-26T08:29:14.6405704Z ##[endgroup] 2024-06-26T08:29:14.6496167Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:29:14.6496628Z with: 2024-06-26T08:29:14.6497059Z s3-bucket: gha-artifacts 2024-06-26T08:29:14.6497504Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:29:14.6498003Z retention-days: 14 2024-06-26T08:29:14.6498356Z if-no-files-found: warn 2024-06-26T08:29:14.6498736Z path: test-jsons-*.zip 2024-06-26T08:29:14.6499100Z name: artifact 2024-06-26T08:29:14.6499424Z region: us-east-1 2024-06-26T08:29:14.6499738Z env: 2024-06-26T08:29:14.6500038Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:14.6500516Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:14.6501256Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:14.6501902Z ##[endgroup] 2024-06-26T08:29:14.9523424Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:29:14.9524245Z With the provided path, there will be 1 file uploaded 2024-06-26T08:29:14.9525095Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:29:14.9549753Z Starting upload of test-jsons-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259.zip 2024-06-26T08:29:15.0959026Z Finished upload of test-jsons-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259.zip 2024-06-26T08:29:15.1119352Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:29:15.1119803Z with: 2024-06-26T08:29:15.1120113Z s3-bucket: gha-artifacts 2024-06-26T08:29:15.1120562Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:29:15.1121056Z retention-days: 14 2024-06-26T08:29:15.1121414Z if-no-files-found: error 2024-06-26T08:29:15.1121805Z path: test-reports-*.zip 2024-06-26T08:29:15.1122180Z name: artifact 2024-06-26T08:29:15.1122521Z region: us-east-1 2024-06-26T08:29:15.1122949Z env: 2024-06-26T08:29:15.1123315Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:15.1123793Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:15.1124541Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:15.1125199Z ##[endgroup] 2024-06-26T08:29:15.4179783Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:29:15.4180741Z With the provided path, there will be 1 file uploaded 2024-06-26T08:29:15.4181485Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:29:15.4206407Z Starting upload of test-reports-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259.zip 2024-06-26T08:29:15.5211958Z Finished upload of test-reports-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259.zip 2024-06-26T08:29:15.5345682Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:29:15.5346138Z with: 2024-06-26T08:29:15.5346445Z s3-bucket: gha-artifacts 2024-06-26T08:29:15.5346890Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:29:15.5347383Z retention-days: 14 2024-06-26T08:29:15.5347738Z if-no-files-found: ignore 2024-06-26T08:29:15.5348126Z path: logs-*.zip 2024-06-26T08:29:15.5348459Z name: artifact 2024-06-26T08:29:15.5348779Z region: us-east-1 2024-06-26T08:29:15.5349115Z env: 2024-06-26T08:29:15.5349418Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:15.5349902Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:15.5350639Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:15.5351289Z ##[endgroup] 2024-06-26T08:29:15.8453602Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:29:15.8454743Z With the provided path, there will be 1 file uploaded 2024-06-26T08:29:15.8455896Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:29:15.8479745Z Starting upload of logs-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259.zip 2024-06-26T08:29:16.0173583Z Finished upload of logs-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_26688911259.zip 2024-06-26T08:29:16.0310051Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:29:16.0310509Z with: 2024-06-26T08:29:16.0310821Z s3-bucket: gha-artifacts 2024-06-26T08:29:16.0311267Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:29:16.0311875Z retention-days: 14 2024-06-26T08:29:16.0312233Z if-no-files-found: ignore 2024-06-26T08:29:16.0312628Z path: debug-*.zip 2024-06-26T08:29:16.0312966Z name: artifact 2024-06-26T08:29:16.0313287Z region: us-east-1 2024-06-26T08:29:16.0313611Z env: 2024-06-26T08:29:16.0313909Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:16.0314376Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:16.0315113Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:16.0315881Z ##[endgroup] 2024-06-26T08:29:16.3298803Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:29:16.3430582Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:29:16.3431239Z # shellcheck disable=SC2156 2024-06-26T08:29:16.3432075Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:29:16.3440761Z shell: /usr/bin/bash -e {0} 2024-06-26T08:29:16.3441148Z env: 2024-06-26T08:29:16.3441453Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:16.3441931Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:16.3442702Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:16.3443375Z ##[endgroup] 2024-06-26T08:29:16.5053638Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:29:16.5054222Z with: 2024-06-26T08:29:16.5054508Z env: 2024-06-26T08:29:16.5054812Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:16.5055284Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:16.5056029Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:16.5056678Z ##[endgroup] 2024-06-26T08:29:16.5074733Z ##[group]Run set -eou pipefail 2024-06-26T08:29:16.5075166Z set -eou pipefail 2024-06-26T08:29:16.5075612Z  2024-06-26T08:29:16.5076152Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:29:16.5076802Z for _ in $(seq 1440); do 2024-06-26T08:29:16.5077458Z  # Break if no ssh session exists anymore 2024-06-26T08:29:16.5077982Z  if [ "$(who)" = "" ]; then 2024-06-26T08:29:16.5078413Z  break 2024-06-26T08:29:16.5078781Z  fi 2024-06-26T08:29:16.5079101Z  echo "." 2024-06-26T08:29:16.5079453Z  sleep 5 2024-06-26T08:29:16.5079796Z done 2024-06-26T08:29:16.5087357Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:16.5087872Z env: 2024-06-26T08:29:16.5088174Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:16.5088647Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:16.5089389Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:16.5090044Z ##[endgroup] 2024-06-26T08:29:16.5109710Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:29:16.5162459Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:29:16.5163227Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:29:16.5163841Z # shellcheck disable=SC2046 2024-06-26T08:29:16.5164328Z docker stop $(docker ps -q) || true 2024-06-26T08:29:16.5164833Z # Prune all of the docker images 2024-06-26T08:29:16.5165296Z docker system prune -af 2024-06-26T08:29:16.5173105Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:16.5173649Z env: 2024-06-26T08:29:16.5173956Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:29:16.5174422Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:29:16.5175169Z DOCKER_CONTAINER_ID: 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:16.5175820Z ##[endgroup] 2024-06-26T08:29:16.8227571Z 60a3066c951d 2024-06-26T08:29:17.5726614Z Deleted Containers: 2024-06-26T08:29:17.5727728Z 60a3066c951d5e83e3810e927e6eadad8b09c22ee549f88a3e554817e1046517 2024-06-26T08:29:17.5728172Z 2024-06-26T08:29:22.7033542Z Deleted Images: 2024-06-26T08:29:22.7035694Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:29:22.7038894Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:29:22.7040850Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:29:22.7041702Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:29:22.7042548Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:29:22.7043420Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:29:22.7044288Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:29:22.7045140Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:29:22.7045974Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:29:22.7047061Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:29:22.7047899Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:29:22.7048706Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:29:22.7049528Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:29:22.7050360Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:29:22.7051185Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:29:22.7052020Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:29:22.7052854Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:29:22.7053681Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:29:22.7054566Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:29:22.7055392Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:29:22.7056229Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:29:22.7057064Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:29:22.7057878Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:29:22.7058702Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:29:22.7059535Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:29:22.7060368Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:29:22.7061213Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:29:22.7062066Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:29:22.7062913Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:29:22.7063755Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:29:22.7064591Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:29:22.7065403Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:29:22.7066208Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:29:22.7067038Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:29:22.7068003Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:29:22.7068832Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:29:22.7069667Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:29:22.7070507Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:29:22.7071334Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:29:22.7072159Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:29:22.7072979Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:29:22.7073857Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:29:22.7074908Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:29:22.7075752Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:29:22.7076563Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:29:22.7077514Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:29:22.7078495Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:29:22.7079355Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:29:22.7080188Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:29:22.7081009Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:29:22.7081832Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:29:22.7082666Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:29:22.7083511Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:29:22.7084375Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:29:22.7085228Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:29:22.7086062Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:29:22.7086905Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:29:22.7087732Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:29:22.7088549Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:29:22.7089376Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:29:22.7090201Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:29:22.7091033Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:29:22.7091872Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:29:22.7092705Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:29:22.7093539Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:29:22.7094380Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:29:22.7095238Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:29:22.7096057Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:29:22.7096892Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:29:22.7097715Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:29:22.7098539Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:29:22.7099452Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:29:22.7100276Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:29:22.7101098Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:29:22.7101938Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:29:22.7102762Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:29:22.7103607Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:29:22.7104504Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:29:22.7105351Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:29:22.7106188Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:29:22.7107031Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:29:22.7107533Z 2024-06-26T08:29:22.7107694Z Total reclaimed space: 36.29GB 2024-06-26T08:29:22.7172080Z Post job cleanup. 2024-06-26T08:29:22.7220271Z Post job cleanup. 2024-06-26T08:29:22.8033881Z [command]/usr/bin/git version 2024-06-26T08:29:22.8076160Z git version 2.40.1 2024-06-26T08:29:22.8116342Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/634ca117-dfc7-4608-8d7d-f8f77c5a1d50' before making global git config changes 2024-06-26T08:29:22.8117761Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:29:22.8122306Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:29:22.8154396Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:29:22.8177251Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:29:22.8399847Z Entering 'android/libs/fbjni' 2024-06-26T08:29:22.8439140Z Entering 'third_party/FP16' 2024-06-26T08:29:22.8477103Z Entering 'third_party/FXdiv' 2024-06-26T08:29:22.8514552Z Entering 'third_party/NNPACK' 2024-06-26T08:29:22.8551586Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:29:22.8588774Z Entering 'third_party/XNNPACK' 2024-06-26T08:29:22.8636640Z Entering 'third_party/benchmark' 2024-06-26T08:29:22.8672230Z Entering 'third_party/cpp-httplib' 2024-06-26T08:29:22.8706246Z Entering 'third_party/cpuinfo' 2024-06-26T08:29:22.8742863Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:29:22.8779056Z Entering 'third_party/cutlass' 2024-06-26T08:29:22.8821735Z Entering 'third_party/eigen' 2024-06-26T08:29:22.8859551Z Entering 'third_party/fbgemm' 2024-06-26T08:29:22.8896418Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:29:22.8931046Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:29:22.8966520Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:29:22.9007706Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:29:22.9043419Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:29:22.9080623Z Entering 'third_party/flatbuffers' 2024-06-26T08:29:22.9118183Z Entering 'third_party/fmt' 2024-06-26T08:29:22.9153522Z Entering 'third_party/foxi' 2024-06-26T08:29:22.9188436Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:29:22.9223378Z Entering 'third_party/gloo' 2024-06-26T08:29:22.9257838Z Entering 'third_party/googletest' 2024-06-26T08:29:22.9290364Z Entering 'third_party/ideep' 2024-06-26T08:29:22.9325868Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:29:22.9365586Z Entering 'third_party/ittapi' 2024-06-26T08:29:22.9402282Z Entering 'third_party/kineto' 2024-06-26T08:29:22.9436512Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:29:22.9471942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:29:22.9509090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:29:22.9543496Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:29:22.9580268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:29:22.9614815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:29:22.9651642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:29:22.9690171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:29:22.9723980Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:29:22.9759721Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:29:22.9795740Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:29:22.9829835Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:29:22.9866473Z Entering 'third_party/mimalloc' 2024-06-26T08:29:22.9900753Z Entering 'third_party/nccl/nccl' 2024-06-26T08:29:22.9935727Z Entering 'third_party/nlohmann' 2024-06-26T08:29:22.9969696Z Entering 'third_party/onnx' 2024-06-26T08:29:23.0019485Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:29:23.0054804Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:29:23.0092385Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:29:23.0126397Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:29:23.0162247Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:29:23.0196958Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:29:23.0229421Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:29:23.0266130Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:29:23.0302341Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:29:23.0337654Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:29:23.0370076Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:29:23.0405433Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:29:23.0441367Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:29:23.0490474Z Entering 'third_party/pocketfft' 2024-06-26T08:29:23.0523482Z Entering 'third_party/protobuf' 2024-06-26T08:29:23.0559788Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:29:23.0594349Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:29:23.0631129Z Entering 'third_party/psimd' 2024-06-26T08:29:23.0666904Z Entering 'third_party/pthreadpool' 2024-06-26T08:29:23.0701312Z Entering 'third_party/pybind11' 2024-06-26T08:29:23.0742133Z Entering 'third_party/python-peachpy' 2024-06-26T08:29:23.0775829Z Entering 'third_party/sleef' 2024-06-26T08:29:23.0807755Z Entering 'third_party/tensorpipe' 2024-06-26T08:29:23.0840875Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:29:23.0875228Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:29:23.0909460Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:29:23.0943534Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:29:23.0977079Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:29:23.1026371Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:29:23.1044728Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1052369Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:29:23.1081125Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:29:23.1286553Z Entering 'android/libs/fbjni' 2024-06-26T08:29:23.1307861Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1331663Z Entering 'third_party/FP16' 2024-06-26T08:29:23.1353196Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1377252Z Entering 'third_party/FXdiv' 2024-06-26T08:29:23.1397625Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1419277Z Entering 'third_party/NNPACK' 2024-06-26T08:29:23.1442083Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1465287Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:29:23.1487129Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1511374Z Entering 'third_party/XNNPACK' 2024-06-26T08:29:23.1533029Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1569497Z Entering 'third_party/benchmark' 2024-06-26T08:29:23.1591795Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1615490Z Entering 'third_party/cpp-httplib' 2024-06-26T08:29:23.1636675Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1660915Z Entering 'third_party/cpuinfo' 2024-06-26T08:29:23.1683425Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1706319Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:29:23.1728986Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1751562Z Entering 'third_party/cutlass' 2024-06-26T08:29:23.1773200Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1803767Z Entering 'third_party/eigen' 2024-06-26T08:29:23.1826331Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1852808Z Entering 'third_party/fbgemm' 2024-06-26T08:29:23.1874406Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1898678Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:29:23.1920094Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1942849Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:29:23.1964021Z http.https://github.com/.extraheader 2024-06-26T08:29:23.1988728Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:29:23.2010189Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2037714Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:29:23.2060147Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2084378Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:29:23.2105104Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2130449Z Entering 'third_party/flatbuffers' 2024-06-26T08:29:23.2153032Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2179733Z Entering 'third_party/fmt' 2024-06-26T08:29:23.2201882Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2225864Z Entering 'third_party/foxi' 2024-06-26T08:29:23.2248108Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2271462Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:29:23.2293642Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2321265Z Entering 'third_party/gloo' 2024-06-26T08:29:23.2342848Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2366611Z Entering 'third_party/googletest' 2024-06-26T08:29:23.2386790Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2409243Z Entering 'third_party/ideep' 2024-06-26T08:29:23.2430819Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2452180Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:29:23.2472748Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2501630Z Entering 'third_party/ittapi' 2024-06-26T08:29:23.2524942Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2547946Z Entering 'third_party/kineto' 2024-06-26T08:29:23.2570290Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2592199Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:29:23.2613902Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2637117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:29:23.2658811Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2682486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:29:23.2704468Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2728126Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:29:23.2748699Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2773115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:29:23.2793043Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2814777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:29:23.2836068Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2860609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:29:23.2882360Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2905407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:29:23.2926232Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2949316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:29:23.2971080Z http.https://github.com/.extraheader 2024-06-26T08:29:23.2994521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:29:23.3015981Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3041461Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:29:23.3063779Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3087054Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:29:23.3108305Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3134219Z Entering 'third_party/mimalloc' 2024-06-26T08:29:23.3155807Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3180193Z Entering 'third_party/nccl/nccl' 2024-06-26T08:29:23.3203511Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3227176Z Entering 'third_party/nlohmann' 2024-06-26T08:29:23.3249207Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3272767Z Entering 'third_party/onnx' 2024-06-26T08:29:23.3295823Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3334058Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:29:23.3355616Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3379540Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:29:23.3402504Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3426934Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:29:23.3450906Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3474611Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:29:23.3496819Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3521199Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:29:23.3542179Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3565651Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:29:23.3586050Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3609908Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:29:23.3630632Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3654414Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:29:23.3675932Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3699304Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:29:23.3721098Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3745047Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:29:23.3765766Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3786891Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:29:23.3808888Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3833933Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:29:23.3856127Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3879945Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:29:23.3901104Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3938688Z Entering 'third_party/pocketfft' 2024-06-26T08:29:23.3961495Z http.https://github.com/.extraheader 2024-06-26T08:29:23.3983682Z Entering 'third_party/protobuf' 2024-06-26T08:29:23.4006285Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4032034Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:29:23.4053540Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4076490Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:29:23.4098402Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4123936Z Entering 'third_party/psimd' 2024-06-26T08:29:23.4146712Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4169336Z Entering 'third_party/pthreadpool' 2024-06-26T08:29:23.4191168Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4216804Z Entering 'third_party/pybind11' 2024-06-26T08:29:23.4237941Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4261081Z Entering 'third_party/python-peachpy' 2024-06-26T08:29:23.4282197Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4304754Z Entering 'third_party/sleef' 2024-06-26T08:29:23.4326430Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4348820Z Entering 'third_party/tensorpipe' 2024-06-26T08:29:23.4371471Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4395201Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:29:23.4417341Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4439848Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:29:23.4461449Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4486076Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:29:23.4507375Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4531056Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:29:23.4551954Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4574442Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:29:23.4595682Z http.https://github.com/.extraheader 2024-06-26T08:29:23.4696115Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:29:23.4713452Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:29:23.4720736Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:29:23.4721274Z ##[endgroup] 2024-06-26T08:29:24.3767699Z Cleaning up orphan processes