2024-06-26T07:01:44.4848476Z Current runner version: '2.317.0' 2024-06-26T07:01:44.4854464Z Runner name: 'i-0c2b88ee687943d7d' 2024-06-26T07:01:44.4855241Z Runner group name: 'Default' 2024-06-26T07:01:44.4856203Z Machine name: 'ip-10-0-68-93' 2024-06-26T07:01:44.4861056Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:01:44.4862945Z Actions: read 2024-06-26T07:01:44.4863445Z Attestations: read 2024-06-26T07:01:44.4863967Z Checks: read 2024-06-26T07:01:44.4864398Z Contents: read 2024-06-26T07:01:44.4865124Z Deployments: read 2024-06-26T07:01:44.4865633Z Discussions: read 2024-06-26T07:01:44.4866116Z Issues: read 2024-06-26T07:01:44.4866556Z Metadata: read 2024-06-26T07:01:44.4867038Z Packages: read 2024-06-26T07:01:44.4867510Z Pages: read 2024-06-26T07:01:44.4868023Z PullRequests: read 2024-06-26T07:01:44.4868521Z RepositoryProjects: read 2024-06-26T07:01:44.4869099Z SecurityEvents: read 2024-06-26T07:01:44.4869616Z Statuses: read 2024-06-26T07:01:44.4870067Z ##[endgroup] 2024-06-26T07:01:44.4873286Z Secret source: Actions 2024-06-26T07:01:44.4874047Z Prepare workflow directory 2024-06-26T07:01:44.8461170Z Prepare all required actions 2024-06-26T07:01:44.8653147Z Getting action download info 2024-06-26T07:01:45.0316285Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:01:45.3389687Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:01:48.1869212Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:01:48.3359631Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:01:48.6264943Z Getting action download info 2024-06-26T07:01:48.7154491Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:01:48.8818197Z Getting action download info 2024-06-26T07:01:48.9736917Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:01:49.1337441Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:01:49.1339577Z ##[group] Inputs 2024-06-26T07:01:49.1340055Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:01:49.1348959Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:01:49.1358420Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:01:49.1359597Z sync-tag: 2024-06-26T07:01:49.1360500Z timeout-minutes: 240 2024-06-26T07:01:49.1360836Z use-gha: 2024-06-26T07:01:49.1361124Z dashboard-tag: 2024-06-26T07:01:49.1361451Z s3-bucket: gha-artifacts 2024-06-26T07:01:49.1361805Z aws-role-to-assume: 2024-06-26T07:01:49.1362144Z ##[endgroup] 2024-06-26T07:01:49.1362995Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:01:49.1932313Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:01:49.2072217Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:01:49.2082779Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:49.2083307Z ##[endgroup] 2024-06-26T07:01:50.2363780Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:01:50.2364337Z with: 2024-06-26T07:01:50.2365061Z github-secret: *** 2024-06-26T07:01:50.2366007Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:01:50.2367006Z activate-with-label: false 2024-06-26T07:01:50.2367363Z label: with-ssh 2024-06-26T07:01:50.2367692Z remove-existing-keys: true 2024-06-26T07:01:50.2368090Z fail-silently: true 2024-06-26T07:01:50.2368403Z env: 2024-06-26T07:01:50.2368667Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:50.2369023Z ##[endgroup] 2024-06-26T07:01:50.3237242Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:01:50.3238257Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:01:50.6546231Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:01:50.7234816Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:01:50.7235625Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:01:50.7908787Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:01:50.8015218Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:01:50.8015881Z with: 2024-06-26T07:01:50.8016216Z submodules: recursive 2024-06-26T07:01:50.8016596Z fetch-depth: 0 2024-06-26T07:01:50.8016929Z env: 2024-06-26T07:01:50.8017237Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:50.8017630Z ##[endgroup] 2024-06-26T07:01:50.8244968Z ##[group]Run retry () { 2024-06-26T07:01:50.8245419Z retry () { 2024-06-26T07:01:50.8246044Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:01:50.8246613Z } 2024-06-26T07:01:50.8246933Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:01:50.8247353Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:01:50.8247841Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:50.8248303Z else 2024-06-26T07:01:50.8248646Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:50.8249079Z fi 2024-06-26T07:01:50.8249423Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:01:50.8257554Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:50.8258071Z env: 2024-06-26T07:01:50.8258357Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:50.8258706Z NO_SUDO: 2024-06-26T07:01:50.8258975Z ##[endgroup] 2024-06-26T07:01:50.8280690Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:53.8470762Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:01:53.8471192Z with: 2024-06-26T07:01:53.8471518Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:53.8471959Z fetch-depth: 0 2024-06-26T07:01:53.8472279Z submodules: recursive 2024-06-26T07:01:53.8472612Z quiet-checkout: true 2024-06-26T07:01:53.8472961Z repository: pytorch/pytorch 2024-06-26T07:01:53.8473459Z token: *** 2024-06-26T07:01:53.8473752Z ssh-strict: true 2024-06-26T07:01:53.8474074Z persist-credentials: true 2024-06-26T07:01:53.8474434Z clean: true 2024-06-26T07:01:53.8474777Z sparse-checkout-cone-mode: true 2024-06-26T07:01:53.8475168Z lfs: false 2024-06-26T07:01:53.8475468Z set-safe-directory: true 2024-06-26T07:01:53.8475810Z env: 2024-06-26T07:01:53.8476092Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:53.8476438Z ##[endgroup] 2024-06-26T07:01:53.9406801Z Syncing repository: pytorch/pytorch 2024-06-26T07:01:53.9408373Z ##[group]Getting Git version info 2024-06-26T07:01:53.9409131Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:53.9409977Z [command]/usr/bin/git version 2024-06-26T07:01:53.9410346Z git version 2.40.1 2024-06-26T07:01:53.9411792Z ##[endgroup] 2024-06-26T07:01:53.9423214Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/9563679b-e46a-4221-a4a0-f699aa5c01b6/.gitconfig' 2024-06-26T07:01:53.9431095Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9563679b-e46a-4221-a4a0-f699aa5c01b6' before making global git config changes 2024-06-26T07:01:53.9432341Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:01:53.9436032Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:53.9470520Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:53.9474538Z ##[group]Initializing the repository 2024-06-26T07:01:53.9477501Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:53.9497117Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:01:53.9498112Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:01:53.9499153Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:01:53.9499727Z hint: 2024-06-26T07:01:53.9500181Z hint: git config --global init.defaultBranch 2024-06-26T07:01:53.9500656Z hint: 2024-06-26T07:01:53.9501351Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:01:53.9502359Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:01:53.9502948Z hint: 2024-06-26T07:01:53.9503264Z hint: git branch -m 2024-06-26T07:01:53.9504009Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:01:53.9505973Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:01:53.9525352Z ##[endgroup] 2024-06-26T07:01:53.9525947Z ##[group]Disabling automatic garbage collection 2024-06-26T07:01:53.9528116Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:01:53.9551968Z ##[endgroup] 2024-06-26T07:01:53.9552548Z ##[group]Setting up auth 2024-06-26T07:01:53.9556592Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:01:53.9581703Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:01:53.9771966Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:01:53.9793821Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:01:53.9988435Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:01:54.0028979Z ##[endgroup] 2024-06-26T07:01:54.0029536Z ##[group]Fetching the repository 2024-06-26T07:01:54.0034606Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:01:56.2845444Z remote: Enumerating objects: 987612 2024-06-26T07:01:56.2846538Z remote: Enumerating objects: 990209, done. 2024-06-26T07:01:56.2849392Z remote: Counting objects: 0% (1/2597) 2024-06-26T07:01:56.2849912Z remote: Counting objects: 1% (26/2597) 2024-06-26T07:01:56.2850907Z remote: Counting objects: 2% (52/2597) 2024-06-26T07:01:56.2852393Z remote: Counting objects: 3% (78/2597) 2024-06-26T07:01:56.2852903Z remote: Counting objects: 4% (104/2597) 2024-06-26T07:01:56.2853441Z remote: Counting objects: 5% (130/2597) 2024-06-26T07:01:56.2856749Z remote: Counting objects: 6% (156/2597) 2024-06-26T07:01:56.2857279Z remote: Counting objects: 7% (182/2597) 2024-06-26T07:01:56.2857784Z remote: Counting objects: 8% (208/2597) 2024-06-26T07:01:56.2858278Z remote: Counting objects: 9% (234/2597) 2024-06-26T07:01:56.2858983Z remote: Counting objects: 10% (260/2597) 2024-06-26T07:01:56.2859524Z remote: Counting objects: 11% (286/2597) 2024-06-26T07:01:56.2860069Z remote: Counting objects: 12% (312/2597) 2024-06-26T07:01:56.2860603Z remote: Counting objects: 13% (338/2597) 2024-06-26T07:01:56.2862569Z remote: Counting objects: 14% (364/2597) 2024-06-26T07:01:56.2863147Z remote: Counting objects: 15% (390/2597) 2024-06-26T07:01:56.2864698Z remote: Counting objects: 16% (416/2597) 2024-06-26T07:01:56.2865215Z remote: Counting objects: 17% (442/2597) 2024-06-26T07:01:56.2865750Z remote: Counting objects: 18% (468/2597) 2024-06-26T07:01:56.2867397Z remote: Counting objects: 19% (494/2597) 2024-06-26T07:01:56.2867959Z remote: Counting objects: 20% (520/2597) 2024-06-26T07:01:56.2868505Z remote: Counting objects: 21% (546/2597) 2024-06-26T07:01:56.2869092Z remote: Counting objects: 22% (572/2597) 2024-06-26T07:01:56.2869633Z remote: Counting objects: 23% (598/2597) 2024-06-26T07:01:56.2870172Z remote: Counting objects: 24% (624/2597) 2024-06-26T07:01:56.2870719Z remote: Counting objects: 25% (650/2597) 2024-06-26T07:01:56.2871560Z remote: Counting objects: 26% (676/2597) 2024-06-26T07:01:56.2872291Z remote: Counting objects: 27% (702/2597) 2024-06-26T07:01:56.2872797Z remote: Counting objects: 28% (728/2597) 2024-06-26T07:01:56.2873304Z remote: Counting objects: 29% (754/2597) 2024-06-26T07:01:56.2874915Z remote: Counting objects: 30% (780/2597) 2024-06-26T07:01:56.2875436Z remote: Counting objects: 31% (806/2597) 2024-06-26T07:01:56.2875951Z remote: Counting objects: 32% (832/2597) 2024-06-26T07:01:56.2876453Z remote: Counting objects: 33% (858/2597) 2024-06-26T07:01:56.2877254Z remote: Counting objects: 34% (883/2597) 2024-06-26T07:01:56.2879986Z remote: Counting objects: 35% (909/2597) 2024-06-26T07:01:56.2880493Z remote: Counting objects: 36% (935/2597) 2024-06-26T07:01:56.2880988Z remote: Counting objects: 37% (961/2597) 2024-06-26T07:01:56.2881482Z remote: Counting objects: 38% (987/2597) 2024-06-26T07:01:56.2882393Z remote: Counting objects: 39% (1013/2597) 2024-06-26T07:01:56.2882927Z remote: Counting objects: 40% (1039/2597) 2024-06-26T07:01:56.2885172Z remote: Counting objects: 41% (1065/2597) 2024-06-26T07:01:56.2885701Z remote: Counting objects: 42% (1091/2597) 2024-06-26T07:01:56.2886220Z remote: Counting objects: 43% (1117/2597) 2024-06-26T07:01:56.2887473Z remote: Counting objects: 44% (1143/2597) 2024-06-26T07:01:56.2888001Z remote: Counting objects: 45% (1169/2597) 2024-06-26T07:01:56.2890207Z remote: Counting objects: 46% (1195/2597) 2024-06-26T07:01:56.2890741Z remote: Counting objects: 47% (1221/2597) 2024-06-26T07:01:56.2891351Z remote: Counting objects: 48% (1247/2597) 2024-06-26T07:01:56.2891874Z remote: Counting objects: 49% (1273/2597) 2024-06-26T07:01:56.2892390Z remote: Counting objects: 50% (1299/2597) 2024-06-26T07:01:56.2892908Z remote: Counting objects: 51% (1325/2597) 2024-06-26T07:01:56.2893418Z remote: Counting objects: 52% (1351/2597) 2024-06-26T07:01:56.2893930Z remote: Counting objects: 53% (1377/2597) 2024-06-26T07:01:56.2894448Z remote: Counting objects: 54% (1403/2597) 2024-06-26T07:01:56.2894961Z remote: Counting objects: 55% (1429/2597) 2024-06-26T07:01:56.2895467Z remote: Counting objects: 56% (1455/2597) 2024-06-26T07:01:56.2895992Z remote: Counting objects: 57% (1481/2597) 2024-06-26T07:01:56.2896506Z remote: Counting objects: 58% (1507/2597) 2024-06-26T07:01:56.2897021Z remote: Counting objects: 59% (1533/2597) 2024-06-26T07:01:56.2897525Z remote: Counting objects: 60% (1559/2597) 2024-06-26T07:01:56.2898039Z remote: Counting objects: 61% (1585/2597) 2024-06-26T07:01:56.2898552Z remote: Counting objects: 62% (1611/2597) 2024-06-26T07:01:56.2899139Z remote: Counting objects: 63% (1637/2597) 2024-06-26T07:01:56.2899660Z remote: Counting objects: 64% (1663/2597) 2024-06-26T07:01:56.2900168Z remote: Counting objects: 65% (1689/2597) 2024-06-26T07:01:56.2900684Z remote: Counting objects: 66% (1715/2597) 2024-06-26T07:01:56.2901193Z remote: Counting objects: 67% (1740/2597) 2024-06-26T07:01:56.2913338Z remote: Counting objects: 68% (1766/2597) 2024-06-26T07:01:56.2913850Z remote: Counting objects: 69% (1792/2597) 2024-06-26T07:01:56.2931160Z remote: Counting objects: 70% (1818/2597) 2024-06-26T07:01:56.2931677Z remote: Counting objects: 71% (1844/2597) 2024-06-26T07:01:56.2932202Z remote: Counting objects: 72% (1870/2597) 2024-06-26T07:01:56.2932726Z remote: Counting objects: 73% (1896/2597) 2024-06-26T07:01:56.2933233Z remote: Counting objects: 74% (1922/2597) 2024-06-26T07:01:56.2933743Z remote: Counting objects: 75% (1948/2597) 2024-06-26T07:01:56.2934259Z remote: Counting objects: 76% (1974/2597) 2024-06-26T07:01:56.2934878Z remote: Counting objects: 77% (2000/2597) 2024-06-26T07:01:56.2935391Z remote: Counting objects: 78% (2026/2597) 2024-06-26T07:01:56.2935915Z remote: Counting objects: 79% (2052/2597) 2024-06-26T07:01:56.2936432Z remote: Counting objects: 80% (2078/2597) 2024-06-26T07:01:56.2936951Z remote: Counting objects: 81% (2104/2597) 2024-06-26T07:01:56.2937463Z remote: Counting objects: 82% (2130/2597) 2024-06-26T07:01:56.2937975Z remote: Counting objects: 83% (2156/2597) 2024-06-26T07:01:56.2938498Z remote: Counting objects: 84% (2182/2597) 2024-06-26T07:01:56.2939008Z remote: Counting objects: 85% (2208/2597) 2024-06-26T07:01:56.2939519Z remote: Counting objects: 86% (2234/2597) 2024-06-26T07:01:56.2940038Z remote: Counting objects: 87% (2260/2597) 2024-06-26T07:01:56.2940560Z remote: Counting objects: 88% (2286/2597) 2024-06-26T07:01:56.2941078Z remote: Counting objects: 89% (2312/2597) 2024-06-26T07:01:56.2941597Z remote: Counting objects: 90% (2338/2597) 2024-06-26T07:01:56.2942114Z remote: Counting objects: 91% (2364/2597) 2024-06-26T07:01:56.2942630Z remote: Counting objects: 92% (2390/2597) 2024-06-26T07:01:56.2943141Z remote: Counting objects: 93% (2416/2597) 2024-06-26T07:01:56.2943651Z remote: Counting objects: 94% (2442/2597) 2024-06-26T07:01:56.2944238Z remote: Counting objects: 95% (2468/2597) 2024-06-26T07:01:56.2944902Z remote: Counting objects: 96% (2494/2597) 2024-06-26T07:01:56.2945426Z remote: Counting objects: 97% (2520/2597) 2024-06-26T07:01:56.2945933Z remote: Counting objects: 98% (2546/2597) 2024-06-26T07:01:56.2946452Z remote: Counting objects: 99% (2572/2597) 2024-06-26T07:01:56.2946968Z remote: Counting objects: 100% (2597/2597) 2024-06-26T07:01:56.2947523Z remote: Counting objects: 100% (2597/2597), done. 2024-06-26T07:01:56.3022760Z remote: Compressing objects: 0% (1/1074) 2024-06-26T07:01:56.3448151Z remote: Compressing objects: 1% (11/1074) 2024-06-26T07:01:56.3866369Z remote: Compressing objects: 2% (22/1074) 2024-06-26T07:01:56.4224712Z remote: Compressing objects: 3% (33/1074) 2024-06-26T07:01:56.4488265Z remote: Compressing objects: 4% (43/1074) 2024-06-26T07:01:56.4741442Z remote: Compressing objects: 5% (54/1074) 2024-06-26T07:01:56.4931745Z remote: Compressing objects: 6% (65/1074) 2024-06-26T07:01:56.5300796Z remote: Compressing objects: 7% (76/1074) 2024-06-26T07:01:56.6151702Z remote: Compressing objects: 8% (86/1074) 2024-06-26T07:01:56.6837677Z remote: Compressing objects: 9% (97/1074) 2024-06-26T07:01:56.7189247Z remote: Compressing objects: 10% (108/1074) 2024-06-26T07:01:56.7219852Z remote: Compressing objects: 11% (119/1074) 2024-06-26T07:01:56.7314012Z remote: Compressing objects: 12% (129/1074) 2024-06-26T07:01:56.7385935Z remote: Compressing objects: 13% (140/1074) 2024-06-26T07:01:56.7424959Z remote: Compressing objects: 14% (151/1074) 2024-06-26T07:01:56.7425502Z remote: Compressing objects: 15% (162/1074) 2024-06-26T07:01:56.7488625Z remote: Compressing objects: 16% (172/1074) 2024-06-26T07:01:56.7492446Z remote: Compressing objects: 17% (183/1074) 2024-06-26T07:01:56.7493532Z remote: Compressing objects: 18% (194/1074) 2024-06-26T07:01:56.7507345Z remote: Compressing objects: 19% (205/1074) 2024-06-26T07:01:56.7527213Z remote: Compressing objects: 20% (215/1074) 2024-06-26T07:01:56.7544988Z remote: Compressing objects: 21% (226/1074) 2024-06-26T07:01:56.7564425Z remote: Compressing objects: 22% (237/1074) 2024-06-26T07:01:56.7582066Z remote: Compressing objects: 23% (248/1074) 2024-06-26T07:01:56.7600993Z remote: Compressing objects: 24% (258/1074) 2024-06-26T07:01:56.7608080Z remote: Compressing objects: 25% (269/1074) 2024-06-26T07:01:56.7614851Z remote: Compressing objects: 26% (280/1074) 2024-06-26T07:01:56.7620110Z remote: Compressing objects: 27% (290/1074) 2024-06-26T07:01:56.7625861Z remote: Compressing objects: 28% (301/1074) 2024-06-26T07:01:56.7634445Z remote: Compressing objects: 29% (312/1074) 2024-06-26T07:01:56.7641921Z remote: Compressing objects: 30% (323/1074) 2024-06-26T07:01:56.7653088Z remote: Compressing objects: 31% (333/1074) 2024-06-26T07:01:56.7655830Z remote: Compressing objects: 32% (344/1074) 2024-06-26T07:01:56.7668753Z remote: Compressing objects: 33% (355/1074) 2024-06-26T07:01:56.7680116Z remote: Compressing objects: 34% (366/1074) 2024-06-26T07:01:56.7696958Z remote: Compressing objects: 35% (376/1074) 2024-06-26T07:01:56.7715164Z remote: Compressing objects: 36% (387/1074) 2024-06-26T07:01:56.7729613Z remote: Compressing objects: 37% (398/1074) 2024-06-26T07:01:56.7745430Z remote: Compressing objects: 38% (409/1074) 2024-06-26T07:01:56.7760746Z remote: Compressing objects: 39% (419/1074) 2024-06-26T07:01:56.7776414Z remote: Compressing objects: 40% (430/1074) 2024-06-26T07:01:56.7796008Z remote: Compressing objects: 41% (441/1074) 2024-06-26T07:01:56.7818139Z remote: Compressing objects: 42% (452/1074) 2024-06-26T07:01:56.7828956Z remote: Compressing objects: 43% (462/1074) 2024-06-26T07:01:56.7831407Z remote: Compressing objects: 44% (473/1074) 2024-06-26T07:01:56.7834439Z remote: Compressing objects: 45% (484/1074) 2024-06-26T07:01:56.7843888Z remote: Compressing objects: 46% (495/1074) 2024-06-26T07:01:56.7855145Z remote: Compressing objects: 47% (505/1074) 2024-06-26T07:01:56.7866381Z remote: Compressing objects: 48% (516/1074) 2024-06-26T07:01:56.7874484Z remote: Compressing objects: 49% (527/1074) 2024-06-26T07:01:56.7882085Z remote: Compressing objects: 50% (537/1074) 2024-06-26T07:01:56.7885639Z remote: Compressing objects: 51% (548/1074) 2024-06-26T07:01:56.7900402Z remote: Compressing objects: 52% (559/1074) 2024-06-26T07:01:56.7909849Z remote: Compressing objects: 53% (570/1074) 2024-06-26T07:01:56.7927960Z remote: Compressing objects: 54% (580/1074) 2024-06-26T07:01:56.7938532Z remote: Compressing objects: 55% (591/1074) 2024-06-26T07:01:56.7948680Z remote: Compressing objects: 56% (602/1074) 2024-06-26T07:01:56.7960255Z remote: Compressing objects: 57% (613/1074) 2024-06-26T07:01:56.7972184Z remote: Compressing objects: 58% (623/1074) 2024-06-26T07:01:56.7983108Z remote: Compressing objects: 59% (634/1074) 2024-06-26T07:01:56.7994691Z remote: Compressing objects: 60% (645/1074) 2024-06-26T07:01:56.8002658Z remote: Compressing objects: 61% (656/1074) 2024-06-26T07:01:56.8016503Z remote: Compressing objects: 62% (666/1074) 2024-06-26T07:01:56.8023148Z remote: Compressing objects: 63% (677/1074) 2024-06-26T07:01:56.8032917Z remote: Compressing objects: 64% (688/1074) 2024-06-26T07:01:56.8040003Z remote: Compressing objects: 65% (699/1074) 2024-06-26T07:01:56.8042637Z remote: Compressing objects: 66% (709/1074) 2024-06-26T07:01:56.8063398Z remote: Compressing objects: 67% (720/1074) 2024-06-26T07:01:56.8066463Z remote: Compressing objects: 68% (731/1074) 2024-06-26T07:01:56.8069173Z remote: Compressing objects: 69% (742/1074) 2024-06-26T07:01:56.8069706Z remote: Compressing objects: 70% (752/1074) 2024-06-26T07:01:56.8070243Z remote: Compressing objects: 71% (763/1074) 2024-06-26T07:01:56.8071643Z remote: Compressing objects: 72% (774/1074) 2024-06-26T07:01:56.8072422Z remote: Compressing objects: 73% (785/1074) 2024-06-26T07:01:56.8073242Z remote: Compressing objects: 74% (795/1074) 2024-06-26T07:01:56.8074187Z remote: Compressing objects: 75% (806/1074) 2024-06-26T07:01:56.8075028Z remote: Compressing objects: 76% (817/1074) 2024-06-26T07:01:56.8075836Z remote: Compressing objects: 77% (827/1074) 2024-06-26T07:01:56.8076659Z remote: Compressing objects: 78% (838/1074) 2024-06-26T07:01:56.8077454Z remote: Compressing objects: 79% (849/1074) 2024-06-26T07:01:56.8080867Z remote: Compressing objects: 80% (860/1074) 2024-06-26T07:01:56.8081432Z remote: Compressing objects: 81% (870/1074) 2024-06-26T07:01:56.8084156Z remote: Compressing objects: 82% (881/1074) 2024-06-26T07:01:56.8084693Z remote: Compressing objects: 83% (892/1074) 2024-06-26T07:01:56.8085238Z remote: Compressing objects: 84% (903/1074) 2024-06-26T07:01:56.8089096Z remote: Compressing objects: 85% (913/1074) 2024-06-26T07:01:56.8100017Z remote: Compressing objects: 86% (924/1074) 2024-06-26T07:01:56.8100652Z remote: Compressing objects: 87% (935/1074) 2024-06-26T07:01:56.8101218Z remote: Compressing objects: 88% (946/1074) 2024-06-26T07:01:56.8101764Z remote: Compressing objects: 89% (956/1074) 2024-06-26T07:01:56.8102304Z remote: Compressing objects: 90% (967/1074) 2024-06-26T07:01:56.8102838Z remote: Compressing objects: 91% (978/1074) 2024-06-26T07:01:56.8103369Z remote: Compressing objects: 92% (989/1074) 2024-06-26T07:01:56.8104042Z remote: Compressing objects: 93% (999/1074) 2024-06-26T07:01:56.8104753Z remote: Compressing objects: 94% (1010/1074) 2024-06-26T07:01:56.8119407Z remote: Compressing objects: 95% (1021/1074) 2024-06-26T07:01:56.8120101Z remote: Compressing objects: 96% (1032/1074) 2024-06-26T07:01:56.8120713Z remote: Compressing objects: 97% (1042/1074) 2024-06-26T07:01:56.8121265Z remote: Compressing objects: 98% (1053/1074) 2024-06-26T07:01:56.8121814Z remote: Compressing objects: 99% (1064/1074) 2024-06-26T07:01:56.8122364Z remote: Compressing objects: 100% (1074/1074) 2024-06-26T07:01:56.8122947Z remote: Compressing objects: 100% (1074/1074), done. 2024-06-26T07:02:16.9861792Z remote: Total 990209 (delta 2062), reused 1863 (delta 1522), pack-reused 987612 2024-06-26T07:02:40.7907274Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:02:40.7924796Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:40.7928467Z ##[endgroup] 2024-06-26T07:02:40.7929029Z ##[group]Determining the checkout info 2024-06-26T07:02:40.7931055Z ##[endgroup] 2024-06-26T07:02:40.7931660Z ##[group]Checking out the ref 2024-06-26T07:02:40.7934170Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:42.0275809Z ##[endgroup] 2024-06-26T07:02:42.0276422Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:02:42.0280506Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:02:42.0324289Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:02:42.0350234Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:02:42.0370849Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:02:42.0392317Z ##[endgroup] 2024-06-26T07:02:42.0392986Z ##[group]Fetching submodules 2024-06-26T07:02:42.0396088Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:02:42.0606126Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:02:42.0809369Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:02:42.0810873Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:02:42.0812644Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:02:42.0814102Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:02:42.0815852Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:42.0817546Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:02:42.0819461Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:02:42.0821878Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:02:42.0823718Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:02:42.0827052Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:02:42.0828925Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:02:42.0831618Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:02:42.0834147Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:02:42.0836651Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:02:42.0839084Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:02:42.0841753Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:02:42.0844610Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:42.0847291Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:02:42.0850216Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:02:42.0853170Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:02:42.0856125Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:02:42.0859179Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:02:42.0862457Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:02:42.0865796Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:02:42.0869544Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:02:42.0872721Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:02:42.0876357Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:02:42.0879610Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:02:42.0883138Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:02:42.0886698Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:02:42.0890516Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:02:42.0894126Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:02:42.0897880Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:02:42.0901618Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:02:42.0905868Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:02:42.0926025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:02:42.4054339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:02:42.6088961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:02:42.7809523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:02:43.0066429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:02:45.0756384Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:02:55.1514055Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:02:55.5878366Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:02:56.0726430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:02:56.6797465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:02:57.8676428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:02:59.6046040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:03:05.4759174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:03:06.7690429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:03:08.5667782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:03:09.7857596Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:03:10.0393879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:03:10.4475562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:03:10.7737786Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:03:11.8780019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:03:12.2557568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:03:12.5395973Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:03:14.2062167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:03:14.9261145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:03:15.5838729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:03:21.5961440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:03:24.0428025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:03:29.7753336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:03:29.9884754Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:03:38.2628690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:03:38.4305573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:03:38.6350741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:03:39.6399811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:03:39.9404330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:03:40.5817465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:03:40.9853045Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:03:40.9941042Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:03:41.0000681Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:03:41.0183128Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:03:41.0474381Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:03:41.8243156Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:03:41.8416804Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:03:41.8760053Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:03:41.9608204Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:03:41.9880244Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:03:42.4265219Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:03:42.6418174Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:03:42.7062243Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:03:42.7073311Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:42.7076367Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:42.7077850Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:42.7079451Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:42.7081112Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:42.7102740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:03:43.7004534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:03:44.3016230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:03:46.0620510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:03:47.0380547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:03:47.3840935Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:03:47.4675734Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:03:47.8104765Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:03:47.8649408Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:03:47.8739490Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:03:47.9662926Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:03:47.9946113Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:03:48.0009215Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:03:48.0313151Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:03:48.0505914Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:03:48.0872985Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:03:48.0967003Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:03:48.0977762Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:48.0993907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:04:00.1101101Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:04:00.1245492Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:04:00.2020419Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:04:00.2031064Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:00.2032886Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:00.2034637Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:00.2052524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:04:00.7340586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:04:01.9580173Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:04:03.0098964Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:04:03.0108970Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:03.0111044Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:03.0113113Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:03.0141693Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:03.0144022Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:03.0146314Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:03.0148476Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:03.0150164Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:03.0151781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:04:04.0529042Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:04:04.4120361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:04:05.6634984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:04:05.9565132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:04:06.4594300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:04:07.4765659Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:04:13.3729999Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:04:13.7474337Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:04:13.7609170Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:04:13.7898675Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:04:13.7992941Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:04:13.8002493Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:13.8017594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:04:14.1438750Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:04:14.1575055Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:04:14.1924129Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:04:14.2716669Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:04:14.2833047Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:04:14.3128479Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:04:14.3615694Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:04:14.3923602Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:04:14.4116329Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:04:14.4975676Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:04:14.7824770Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:04:14.7853949Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:14.7855734Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:14.7872413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:04:15.2383523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:04:16.3807126Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:04:16.4083838Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:04:16.4614351Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:04:16.4627135Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:16.4633043Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:16.4634650Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:16.4636256Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:16.4638730Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:16.4640936Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:16.4642863Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:16.4644444Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:16.4659905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:04:16.8422781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:04:17.8276629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:04:18.1250705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:04:23.9032919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:04:24.1934736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:04:24.3817261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:04:24.6812967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:04:32.1815743Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:04:32.2146754Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:04:32.2272080Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:04:32.3106848Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:04:32.3203884Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:04:32.3311507Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:04:32.3426202Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:04:32.3436691Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:32.3438958Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:32.3456231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:04:34.1614156Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:04:35.3661826Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:04:35.4029720Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:04:35.7663127Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:04:35.7754250Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:04:35.9935867Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:04:35.9949597Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:35.9951339Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:35.9968557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:04:36.3716223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:04:37.3739684Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:04:37.4368827Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:04:37.4426984Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:04:37.4514326Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:04:37.4791679Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:04:37.5013112Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:04:37.5355364Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:04:37.5555807Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:04:37.5567331Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:37.5569195Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:37.5570679Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:37.5572795Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:37.5591089Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:04:38.5238713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:04:38.7611875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:04:39.7736084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:04:40.8589182Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:04:40.8702443Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:04:40.9272074Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:04:40.9521777Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:04:40.9531208Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:40.9546959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:04:41.1351796Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:04:41.1374747Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:04:41.1577787Z Entering 'android/libs/fbjni' 2024-06-26T07:04:41.1606102Z Entering 'third_party/FP16' 2024-06-26T07:04:41.1634194Z Entering 'third_party/FXdiv' 2024-06-26T07:04:41.1659810Z Entering 'third_party/NNPACK' 2024-06-26T07:04:41.1689376Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:41.1715825Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:41.1755495Z Entering 'third_party/benchmark' 2024-06-26T07:04:41.1781736Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:41.1808413Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:41.1839097Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:41.1866055Z Entering 'third_party/cutlass' 2024-06-26T07:04:41.1899415Z Entering 'third_party/eigen' 2024-06-26T07:04:41.1931187Z Entering 'third_party/fbgemm' 2024-06-26T07:04:41.1959226Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:41.1987764Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:41.2019542Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:41.2051431Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:41.2079756Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:41.2106618Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:41.2135589Z Entering 'third_party/fmt' 2024-06-26T07:04:41.2165577Z Entering 'third_party/foxi' 2024-06-26T07:04:41.2193511Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:41.2222954Z Entering 'third_party/gloo' 2024-06-26T07:04:41.2252065Z Entering 'third_party/googletest' 2024-06-26T07:04:41.2279400Z Entering 'third_party/ideep' 2024-06-26T07:04:41.2309093Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:41.2342786Z Entering 'third_party/ittapi' 2024-06-26T07:04:41.2373038Z Entering 'third_party/kineto' 2024-06-26T07:04:41.2399956Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:41.2427603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:41.2455891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:41.2484724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:41.2512766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:41.2539292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:41.2568961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:41.2596245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:41.2623186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:41.2651703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:41.2681949Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:41.2709867Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:41.2738312Z Entering 'third_party/mimalloc' 2024-06-26T07:04:41.2767233Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:41.2794007Z Entering 'third_party/nlohmann' 2024-06-26T07:04:41.2820836Z Entering 'third_party/onnx' 2024-06-26T07:04:41.2863086Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:41.2891698Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:41.2920759Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:41.2951533Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:41.2978186Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:41.3004173Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:41.3032061Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:41.3059553Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:41.3088306Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:41.3114191Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:41.3140926Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:41.3167983Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:41.3195866Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:41.3238981Z Entering 'third_party/pocketfft' 2024-06-26T07:04:41.3267541Z Entering 'third_party/protobuf' 2024-06-26T07:04:41.3296542Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:41.3324283Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:41.3351124Z Entering 'third_party/psimd' 2024-06-26T07:04:41.3378946Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:41.3407095Z Entering 'third_party/pybind11' 2024-06-26T07:04:41.3435229Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:41.3463063Z Entering 'third_party/sleef' 2024-06-26T07:04:41.3491777Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:41.3518526Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:41.3546223Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:41.3572486Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:41.3601747Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:41.3627733Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:41.3667442Z ##[endgroup] 2024-06-26T07:04:41.3670494Z ##[group]Persisting credentials for submodules 2024-06-26T07:04:41.3672470Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:04:41.3874916Z Entering 'android/libs/fbjni' 2024-06-26T07:04:41.3907563Z Entering 'third_party/FP16' 2024-06-26T07:04:41.3939699Z Entering 'third_party/FXdiv' 2024-06-26T07:04:41.3974728Z Entering 'third_party/NNPACK' 2024-06-26T07:04:41.4009874Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:41.4045584Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:41.4091631Z Entering 'third_party/benchmark' 2024-06-26T07:04:41.4127596Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:41.4163650Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:41.4198119Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:41.4234328Z Entering 'third_party/cutlass' 2024-06-26T07:04:41.4275084Z Entering 'third_party/eigen' 2024-06-26T07:04:41.4312059Z Entering 'third_party/fbgemm' 2024-06-26T07:04:41.4346490Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:41.4380260Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:41.4415802Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:41.4456508Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:41.4491650Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:41.4527905Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:41.4565863Z Entering 'third_party/fmt' 2024-06-26T07:04:41.4599074Z Entering 'third_party/foxi' 2024-06-26T07:04:41.4635490Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:41.4668251Z Entering 'third_party/gloo' 2024-06-26T07:04:41.4703426Z Entering 'third_party/googletest' 2024-06-26T07:04:41.4739174Z Entering 'third_party/ideep' 2024-06-26T07:04:41.4772907Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:41.4812097Z Entering 'third_party/ittapi' 2024-06-26T07:04:41.4847015Z Entering 'third_party/kineto' 2024-06-26T07:04:41.4883954Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:41.4919752Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:41.4954024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:41.4988500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:41.5020339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:41.5055182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:41.5090452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:41.5125186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:41.5160969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:41.5195158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:41.5230700Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:41.5264973Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:41.5301078Z Entering 'third_party/mimalloc' 2024-06-26T07:04:41.5335726Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:41.5369292Z Entering 'third_party/nlohmann' 2024-06-26T07:04:41.5402298Z Entering 'third_party/onnx' 2024-06-26T07:04:41.5448030Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:41.5483720Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:41.5521289Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:41.5555006Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:41.5590703Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:41.5623233Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:41.5656883Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:41.5690985Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:41.5725175Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:41.5759200Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:41.5792102Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:41.5825481Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:41.5860936Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:41.5911516Z Entering 'third_party/pocketfft' 2024-06-26T07:04:41.5946201Z Entering 'third_party/protobuf' 2024-06-26T07:04:41.5984677Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:41.6020806Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:41.6055402Z Entering 'third_party/psimd' 2024-06-26T07:04:41.6089971Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:41.6124905Z Entering 'third_party/pybind11' 2024-06-26T07:04:41.6160195Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:41.6195305Z Entering 'third_party/sleef' 2024-06-26T07:04:41.6227747Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:41.6260766Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:41.6294994Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:41.6331037Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:41.6364104Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:41.6397187Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:41.6446385Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:04:41.6653120Z Entering 'android/libs/fbjni' 2024-06-26T07:04:41.6685062Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:04:41.6697454Z Entering 'third_party/FP16' 2024-06-26T07:04:41.6729956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:04:41.6742056Z Entering 'third_party/FXdiv' 2024-06-26T07:04:41.6772762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:04:41.6786103Z Entering 'third_party/NNPACK' 2024-06-26T07:04:41.6815699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:04:41.6828710Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:41.6859786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:04:41.6872179Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:41.6903069Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:04:41.6927401Z Entering 'third_party/benchmark' 2024-06-26T07:04:41.6958116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:41.6971936Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:41.7005024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:04:41.7017973Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:41.7050990Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:41.7066609Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:41.7096943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:04:41.7110334Z Entering 'third_party/cutlass' 2024-06-26T07:04:41.7142567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:41.7160609Z Entering 'third_party/eigen' 2024-06-26T07:04:41.7192125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:04:41.7205726Z Entering 'third_party/fbgemm' 2024-06-26T07:04:41.7238756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:04:41.7251517Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:41.7282777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:04:41.7295328Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:41.7327116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:41.7339422Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:41.7372091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:41.7390948Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:41.7421146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:41.7433673Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:41.7465340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:04:41.7479420Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:41.7510672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:04:41.7524285Z Entering 'third_party/fmt' 2024-06-26T07:04:41.7555011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:41.7566615Z Entering 'third_party/foxi' 2024-06-26T07:04:41.7599613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:04:41.7613760Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:41.7646095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:04:41.7658392Z Entering 'third_party/gloo' 2024-06-26T07:04:41.7688784Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:04:41.7702632Z Entering 'third_party/googletest' 2024-06-26T07:04:41.7735826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:41.7750310Z Entering 'third_party/ideep' 2024-06-26T07:04:41.7783022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:04:41.7794183Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:41.7826239Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:04:41.7844054Z Entering 'third_party/ittapi' 2024-06-26T07:04:41.7876697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:04:41.7888348Z Entering 'third_party/kineto' 2024-06-26T07:04:41.7918747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:04:41.7932098Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:41.7964378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:04:41.7977161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:41.8010369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:04:41.8024060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:41.8056094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:04:41.8069469Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:41.8101033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:41.8113083Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:41.8143788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:04:41.8155969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:41.8188927Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:04:41.8202527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:41.8234473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:04:41.8247030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:41.8279058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:41.8291112Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:41.8323832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:04:41.8336823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:41.8369322Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:04:41.8383161Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:41.8415815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:04:41.8429614Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:41.8459570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:04:41.8472359Z Entering 'third_party/mimalloc' 2024-06-26T07:04:41.8503783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:04:41.8515155Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:41.8546924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:04:41.8560938Z Entering 'third_party/nlohmann' 2024-06-26T07:04:41.8591703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:04:41.8606120Z Entering 'third_party/onnx' 2024-06-26T07:04:41.8637396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:04:41.8663055Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:41.8695316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:41.8707789Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:41.8738720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:41.8754007Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:41.8786390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:04:41.8799642Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:41.8831062Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:41.8844000Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:41.8875719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:41.8888374Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:41.8919341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:04:41.8930008Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:41.8962381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:04:41.8975731Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:41.9007309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:04:41.9019672Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:41.9051578Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:04:41.9062796Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:41.9095393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:04:41.9107427Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:41.9138092Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:04:41.9152751Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:41.9183672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:04:41.9197616Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:41.9227757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:04:41.9254964Z Entering 'third_party/pocketfft' 2024-06-26T07:04:41.9287168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:04:41.9298161Z Entering 'third_party/protobuf' 2024-06-26T07:04:41.9329205Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:04:41.9345008Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:41.9376399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:41.9388787Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:41.9420777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:41.9436370Z Entering 'third_party/psimd' 2024-06-26T07:04:41.9469052Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:04:41.9481843Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:41.9513394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:04:41.9526378Z Entering 'third_party/pybind11' 2024-06-26T07:04:41.9556908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:41.9570802Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:41.9602019Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:04:41.9615967Z Entering 'third_party/sleef' 2024-06-26T07:04:41.9648372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:04:41.9661307Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:41.9691087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:04:41.9703199Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:41.9735384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:41.9748849Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:41.9778519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:04:41.9791032Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:41.9822320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:04:41.9835040Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:41.9867993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:41.9879671Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:41.9911438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:04:42.0841974Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:04:42.1044869Z Entering 'android/libs/fbjni' 2024-06-26T07:04:42.1071089Z Entering 'third_party/FP16' 2024-06-26T07:04:42.1096147Z Entering 'third_party/FXdiv' 2024-06-26T07:04:42.1126369Z Entering 'third_party/NNPACK' 2024-06-26T07:04:42.1151902Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:42.1177237Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:42.1217409Z Entering 'third_party/benchmark' 2024-06-26T07:04:42.1247439Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:42.1277571Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:42.1307491Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:42.1333249Z Entering 'third_party/cutlass' 2024-06-26T07:04:42.1367079Z Entering 'third_party/eigen' 2024-06-26T07:04:42.1394753Z Entering 'third_party/fbgemm' 2024-06-26T07:04:42.1423294Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:42.1448699Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:42.1475698Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:42.1509274Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:42.1536581Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:42.1565440Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:42.1596050Z Entering 'third_party/fmt' 2024-06-26T07:04:42.1624024Z Entering 'third_party/foxi' 2024-06-26T07:04:42.1652468Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:42.1680952Z Entering 'third_party/gloo' 2024-06-26T07:04:42.1708940Z Entering 'third_party/googletest' 2024-06-26T07:04:42.1735416Z Entering 'third_party/ideep' 2024-06-26T07:04:42.1761685Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:42.1792982Z Entering 'third_party/ittapi' 2024-06-26T07:04:42.1820345Z Entering 'third_party/kineto' 2024-06-26T07:04:42.1848580Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:42.1875650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:42.1904093Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:42.1932663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:42.1959081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:42.1983014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:42.2012449Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:42.2038432Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:42.2066923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:42.2095382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:42.2124394Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:42.2150967Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:42.2180417Z Entering 'third_party/mimalloc' 2024-06-26T07:04:42.2210697Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:42.2239117Z Entering 'third_party/nlohmann' 2024-06-26T07:04:42.2267678Z Entering 'third_party/onnx' 2024-06-26T07:04:42.2308599Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:42.2336143Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:42.2366101Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:42.2395720Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:42.2423280Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:42.2449844Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:42.2475835Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:42.2503857Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:42.2529508Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:42.2555623Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:42.2582900Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:42.2612551Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:42.2642236Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:42.2685141Z Entering 'third_party/pocketfft' 2024-06-26T07:04:42.2713662Z Entering 'third_party/protobuf' 2024-06-26T07:04:42.2745083Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:42.2772731Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:42.2801821Z Entering 'third_party/psimd' 2024-06-26T07:04:42.2828991Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:42.2857711Z Entering 'third_party/pybind11' 2024-06-26T07:04:42.2886013Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:42.2912705Z Entering 'third_party/sleef' 2024-06-26T07:04:42.2939832Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:42.2968751Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:42.2997377Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:42.3022953Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:42.3051093Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:42.3076316Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:42.3117872Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:04:42.3318925Z Entering 'android/libs/fbjni' 2024-06-26T07:04:42.3344838Z Entering 'third_party/FP16' 2024-06-26T07:04:42.3373703Z Entering 'third_party/FXdiv' 2024-06-26T07:04:42.3400781Z Entering 'third_party/NNPACK' 2024-06-26T07:04:42.3429486Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:42.3459803Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:42.3497336Z Entering 'third_party/benchmark' 2024-06-26T07:04:42.3524941Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:42.3552639Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:42.3580389Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:42.3609969Z Entering 'third_party/cutlass' 2024-06-26T07:04:42.3642105Z Entering 'third_party/eigen' 2024-06-26T07:04:42.3674244Z Entering 'third_party/fbgemm' 2024-06-26T07:04:42.3704108Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:42.3731909Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:42.3758464Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:42.3789326Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:42.3815366Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:42.3845504Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:42.3874629Z Entering 'third_party/fmt' 2024-06-26T07:04:42.3902084Z Entering 'third_party/foxi' 2024-06-26T07:04:42.3929738Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:42.3960026Z Entering 'third_party/gloo' 2024-06-26T07:04:42.3987055Z Entering 'third_party/googletest' 2024-06-26T07:04:42.4012992Z Entering 'third_party/ideep' 2024-06-26T07:04:42.4038786Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:42.4069993Z Entering 'third_party/ittapi' 2024-06-26T07:04:42.4096430Z Entering 'third_party/kineto' 2024-06-26T07:04:42.4124778Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:42.4151563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:42.4179207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:42.4206941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:42.4233986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:42.4261161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:42.4290031Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:42.4319830Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:42.4348089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:42.4376310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:42.4404136Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:42.4430973Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:42.4459728Z Entering 'third_party/mimalloc' 2024-06-26T07:04:42.4489046Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:42.4518183Z Entering 'third_party/nlohmann' 2024-06-26T07:04:42.4546014Z Entering 'third_party/onnx' 2024-06-26T07:04:42.4582711Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:42.4610789Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:42.4641983Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:42.4671255Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:42.4698489Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:42.4726310Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:42.4753484Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:42.4780659Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:42.4808463Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:42.4837012Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:42.4863593Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:42.4891551Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:42.4920190Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:42.4961841Z Entering 'third_party/pocketfft' 2024-06-26T07:04:42.4992182Z Entering 'third_party/protobuf' 2024-06-26T07:04:42.5022089Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:42.5050591Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:42.5079203Z Entering 'third_party/psimd' 2024-06-26T07:04:42.5107879Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:42.5134612Z Entering 'third_party/pybind11' 2024-06-26T07:04:42.5162449Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:42.5191755Z Entering 'third_party/sleef' 2024-06-26T07:04:42.5217343Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:42.5243946Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:42.5269882Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:42.5297532Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:42.5326624Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:42.5352047Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:42.5392906Z ##[endgroup] 2024-06-26T07:04:42.5421617Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:04:42.5443313Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:04:42.5615635Z Prepare all required actions 2024-06-26T07:04:42.5616091Z Getting action download info 2024-06-26T07:04:42.7228573Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:04:42.7229017Z env: 2024-06-26T07:04:42.7229294Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.7229648Z ##[endgroup] 2024-06-26T07:04:42.7289032Z ##[group]Run set -euo pipefail 2024-06-26T07:04:42.7289472Z set -euo pipefail 2024-06-26T07:04:42.7289870Z function get_ec2_metadata() { 2024-06-26T07:04:42.7290416Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:04:42.7291376Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:04:42.7292158Z  category=$1 2024-06-26T07:04:42.7292690Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:04:42.7293334Z  runner_name_str=i-0c2b88ee687943d7d 2024-06-26T07:04:42.7293832Z  if [[ -f /.inarc ]]; then 2024-06-26T07:04:42.7294333Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:04:42.7294911Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:04:42.7295602Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:04:42.7296213Z  else 2024-06-26T07:04:42.7296715Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:04:42.7297299Z  fi 2024-06-26T07:04:42.7297592Z } 2024-06-26T07:04:42.7298038Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:04:42.7298681Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:04:42.7299354Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:04:42.7299921Z echo "system info $(uname -a)" 2024-06-26T07:04:42.7307992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.7308540Z env: 2024-06-26T07:04:42.7308825Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.7309184Z ##[endgroup] 2024-06-26T07:04:42.7382095Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:04:42.7428105Z instance-id: i-0c2b88ee687943d7d 2024-06-26T07:04:42.7474894Z instance-type: g5.4xlarge 2024-06-26T07:04:42.7481420Z system info Linux ip-10-0-68-93.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:04:42.7501395Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:42.7502339Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:42.7510288Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.7510794Z env: 2024-06-26T07:04:42.7511083Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.7511438Z ##[endgroup] 2024-06-26T07:04:42.7574167Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:04:42.7574783Z if systemctl is-active --quiet docker; then 2024-06-26T07:04:42.7575325Z  echo "Docker daemon is running..."; 2024-06-26T07:04:42.7575779Z else 2024-06-26T07:04:42.7576276Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:04:42.7576867Z fi 2024-06-26T07:04:42.7584931Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.7585440Z env: 2024-06-26T07:04:42.7585718Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.7586067Z ##[endgroup] 2024-06-26T07:04:42.7621856Z Docker daemon is running... 2024-06-26T07:04:42.7665661Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:04:42.7666240Z with: 2024-06-26T07:04:42.7666513Z shell: bash 2024-06-26T07:04:42.7666803Z timeout_minutes: 5 2024-06-26T07:04:42.7667129Z max_attempts: 3 2024-06-26T07:04:42.7667448Z retry_wait_seconds: 30 2024-06-26T07:04:42.7669024Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:04:42.7670478Z polling_interval_seconds: 1 2024-06-26T07:04:42.7670864Z warning_on_retry: true 2024-06-26T07:04:42.7671219Z continue_on_error: false 2024-06-26T07:04:42.7671567Z env: 2024-06-26T07:04:42.7671835Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.7672200Z AWS_RETRY_MODE: standard 2024-06-26T07:04:42.7672563Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:04:42.7672912Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:04:42.7673287Z ##[endgroup] 2024-06-26T07:04:43.6297916Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:43.6298860Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:43.6299702Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:43.6300247Z 2024-06-26T07:04:43.6300397Z Login Succeeded 2024-06-26T07:04:43.8175525Z Command completed after 1 attempt(s). 2024-06-26T07:04:43.8218515Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:43.8219233Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:43.8219900Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:43.8228201Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.8228711Z env: 2024-06-26T07:04:43.8228995Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.8229348Z ##[endgroup] 2024-06-26T07:04:43.8291214Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:43.8291998Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:43.8292599Z # shellcheck disable=SC2046 2024-06-26T07:04:43.8293072Z docker stop $(docker ps -q) || true 2024-06-26T07:04:43.8293570Z # Prune all of the docker images 2024-06-26T07:04:43.8294032Z docker system prune -af 2024-06-26T07:04:43.8301157Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.8301662Z env: 2024-06-26T07:04:43.8301942Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.8302295Z ##[endgroup] 2024-06-26T07:04:43.8548858Z "docker stop" requires at least 1 argument. 2024-06-26T07:04:43.8549572Z See 'docker stop --help'. 2024-06-26T07:04:43.8549824Z 2024-06-26T07:04:43.8550048Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:04:43.8550423Z 2024-06-26T07:04:43.8550572Z Stop one or more running containers 2024-06-26T07:04:43.8696897Z Total reclaimed space: 0B 2024-06-26T07:04:43.8727055Z ##[group]Run set +e 2024-06-26T07:04:43.8727411Z set +e 2024-06-26T07:04:43.8727711Z set -x 2024-06-26T07:04:43.8728002Z  2024-06-26T07:04:43.8728319Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:43.8729126Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:04:43.8730216Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:04:43.8731086Z # one is returned at random 2024-06-26T07:04:43.8731635Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:04:43.8732140Z  2024-06-26T07:04:43.8732454Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:43.8733080Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:04:43.8733852Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:04:43.8734417Z  2024-06-26T07:04:43.8734745Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:43.8735302Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:04:43.8735819Z  exit 1 2024-06-26T07:04:43.8736127Z  fi 2024-06-26T07:04:43.8736412Z fi 2024-06-26T07:04:43.8736812Z  2024-06-26T07:04:43.8737179Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:04:43.8737701Z  # Clean up any old records first 2024-06-26T07:04:43.8738222Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:04:43.8738678Z fi 2024-06-26T07:04:43.8738971Z  2024-06-26T07:04:43.8739436Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:04:43.8739981Z cat /etc/hosts 2024-06-26T07:04:43.8747997Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.8748492Z env: 2024-06-26T07:04:43.8748776Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.8749127Z ##[endgroup] 2024-06-26T07:04:43.8768665Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:43.8772506Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:04:43.8773443Z ++ tail -n1 2024-06-26T07:04:43.9116300Z + RESOLVED_IP=18.160.10.28 2024-06-26T07:04:43.9116814Z + '[' -z 18.160.10.28 ']' 2024-06-26T07:04:43.9117275Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:04:43.9122898Z 18.67.76.42 download.pytorch.org 2024-06-26T07:04:43.9123789Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:04:43.9221710Z + echo '18.160.10.28 download.pytorch.org' 2024-06-26T07:04:43.9222462Z + sudo tee -a /etc/hosts 2024-06-26T07:04:43.9457880Z 18.160.10.28 download.pytorch.org 2024-06-26T07:04:43.9470392Z + cat /etc/hosts 2024-06-26T07:04:43.9475034Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:04:43.9487045Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:04:43.9487579Z 18.160.10.28 download.pytorch.org 2024-06-26T07:04:43.9696796Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:04:43.9697398Z with: 2024-06-26T07:04:43.9698492Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9699710Z docker-build-dir: .ci/docker 2024-06-26T07:04:43.9700082Z working-directory: . 2024-06-26T07:04:43.9700543Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.9701067Z force-push: false 2024-06-26T07:04:43.9701356Z env: 2024-06-26T07:04:43.9701624Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.9701960Z ##[endgroup] 2024-06-26T07:04:43.9719246Z ##[group]Run set -ex 2024-06-26T07:04:43.9719622Z set -ex 2024-06-26T07:04:43.9719924Z  2024-06-26T07:04:43.9720495Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:04:43.9721622Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:04:43.9722424Z # job could then download the pre-built image as usual 2024-06-26T07:04:43.9723179Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:04:43.9723862Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9724523Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9725099Z  2024-06-26T07:04:43.9725623Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:04:43.9726267Z  exit 0 2024-06-26T07:04:43.9726582Z else 2024-06-26T07:04:43.9726958Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9727418Z fi 2024-06-26T07:04:43.9727713Z  2024-06-26T07:04:43.9728204Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:04:43.9729111Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:04:43.9729917Z  # use it as it is, but first let's extract the tag 2024-06-26T07:04:43.9730638Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:04:43.9731443Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9732152Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9732710Z else 2024-06-26T07:04:43.9733146Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:04:43.9733818Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9734718Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9735462Z fi 2024-06-26T07:04:43.9743296Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.9743796Z env: 2024-06-26T07:04:43.9744074Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.9744606Z REPO_NAME: pytorch 2024-06-26T07:04:43.9745766Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9746960Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:43.9747460Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.9747980Z ##[endgroup] 2024-06-26T07:04:43.9767294Z + [[ ! -d .ci/docker ]] 2024-06-26T07:04:43.9767726Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:04:43.9768114Z + echo skip=false 2024-06-26T07:04:43.9770160Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:04:43.9773192Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9774404Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:04:43.9779948Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9780617Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9782111Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9807007Z ##[group]Run set +e 2024-06-26T07:04:43.9807376Z set +e 2024-06-26T07:04:43.9807679Z set -x 2024-06-26T07:04:43.9807965Z  2024-06-26T07:04:43.9808243Z login() { 2024-06-26T07:04:43.9808920Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:43.9809800Z } 2024-06-26T07:04:43.9810073Z  2024-06-26T07:04:43.9810353Z retry () { 2024-06-26T07:04:43.9810753Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:43.9811283Z } 2024-06-26T07:04:43.9811558Z  2024-06-26T07:04:43.9811895Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:43.9812317Z  2024-06-26T07:04:43.9812787Z # Check if image already exists, if it does then skip building it 2024-06-26T07:04:43.9813502Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:04:43.9814010Z  exit 0 2024-06-26T07:04:43.9814320Z fi 2024-06-26T07:04:43.9814592Z  2024-06-26T07:04:43.9815104Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:04:43.9815938Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:04:43.9816704Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:04:43.9817381Z  # if we're on the base branch then use the parent commit 2024-06-26T07:04:43.9817991Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:04:43.9818430Z else 2024-06-26T07:04:43.9818902Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:04:43.9819629Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:04:43.9820147Z fi 2024-06-26T07:04:43.9820432Z  2024-06-26T07:04:43.9820746Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:04:43.9821255Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9821726Z  2024-06-26T07:04:43.9822386Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:04:43.9823147Z  exit 0 2024-06-26T07:04:43.9823440Z fi 2024-06-26T07:04:43.9823727Z  2024-06-26T07:04:43.9824174Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:04:43.9825385Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:04:43.9826184Z  exit 1 2024-06-26T07:04:43.9826492Z fi 2024-06-26T07:04:43.9826773Z  2024-06-26T07:04:43.9827265Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:04:43.9828223Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:04:43.9829109Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:04:43.9830129Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:04:43.9831241Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:04:43.9832019Z fi 2024-06-26T07:04:43.9832304Z  2024-06-26T07:04:43.9832670Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:43.9839532Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.9840052Z env: 2024-06-26T07:04:43.9840337Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.9840711Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:43.9841181Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:43.9842464Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9843716Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.9844345Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.9844885Z ##[endgroup] 2024-06-26T07:04:43.9864129Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.9865228Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.9866064Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:43.9866837Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:44.3980889Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:44.3981952Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:44.3982919Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:44.3983438Z 2024-06-26T07:04:44.3983570Z Login Succeeded 2024-06-26T07:04:44.3993117Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:44.5956720Z { 2024-06-26T07:04:44.5957105Z "schemaVersion": 2, 2024-06-26T07:04:44.5957911Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:04:44.5958693Z "config": { 2024-06-26T07:04:44.5959321Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:04:44.5960424Z "size": 48681, 2024-06-26T07:04:44.5961164Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:04:44.5962016Z }, 2024-06-26T07:04:44.5962348Z "layers": [ 2024-06-26T07:04:44.5962708Z { 2024-06-26T07:04:44.5963316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5964041Z "size": 28580681, 2024-06-26T07:04:44.5964827Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:04:44.5965716Z }, 2024-06-26T07:04:44.5966067Z { 2024-06-26T07:04:44.5966654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5967484Z "size": 7943451, 2024-06-26T07:04:44.5968264Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:04:44.5969160Z }, 2024-06-26T07:04:44.5969501Z { 2024-06-26T07:04:44.5970055Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5970991Z "size": 55728572, 2024-06-26T07:04:44.5971555Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:04:44.5972398Z }, 2024-06-26T07:04:44.5972697Z { 2024-06-26T07:04:44.5973292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5973973Z "size": 186, 2024-06-26T07:04:44.5974514Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:04:44.5975133Z }, 2024-06-26T07:04:44.5975382Z { 2024-06-26T07:04:44.5975808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5976367Z "size": 6886, 2024-06-26T07:04:44.5976910Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:04:44.5977547Z }, 2024-06-26T07:04:44.5977979Z { 2024-06-26T07:04:44.5978420Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5978977Z "size": 1329001756, 2024-06-26T07:04:44.5979307Z + exit 0 2024-06-26T07:04:44.5979874Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:04:44.5980496Z }, 2024-06-26T07:04:44.5980747Z { 2024-06-26T07:04:44.5981173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5981714Z "size": 62501, 2024-06-26T07:04:44.5982264Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:04:44.5982865Z }, 2024-06-26T07:04:44.5983109Z { 2024-06-26T07:04:44.5983531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5984084Z "size": 1684, 2024-06-26T07:04:44.5984932Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:04:44.5988213Z }, 2024-06-26T07:04:44.5988573Z { 2024-06-26T07:04:44.5989168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5989880Z "size": 1523, 2024-06-26T07:04:44.5990579Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:04:44.5991378Z }, 2024-06-26T07:04:44.5991690Z { 2024-06-26T07:04:44.5992237Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5992942Z "size": 2528295403, 2024-06-26T07:04:44.5993684Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:04:44.5994557Z }, 2024-06-26T07:04:44.5994878Z { 2024-06-26T07:04:44.5995399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5995991Z "size": 86016, 2024-06-26T07:04:44.5996737Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:04:44.5997536Z }, 2024-06-26T07:04:44.5997871Z { 2024-06-26T07:04:44.5998316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.5998999Z "size": 1823, 2024-06-26T07:04:44.5999556Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:04:44.6000190Z }, 2024-06-26T07:04:44.6000443Z { 2024-06-26T07:04:44.6000875Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6001436Z "size": 246737542, 2024-06-26T07:04:44.6001999Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:04:44.6002613Z }, 2024-06-26T07:04:44.6002852Z { 2024-06-26T07:04:44.6003289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6003835Z "size": 545, 2024-06-26T07:04:44.6004370Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:04:44.6004991Z }, 2024-06-26T07:04:44.6005241Z { 2024-06-26T07:04:44.6005685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6006230Z "size": 1283, 2024-06-26T07:04:44.6006773Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:04:44.6007377Z }, 2024-06-26T07:04:44.6007627Z { 2024-06-26T07:04:44.6008046Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6008608Z "size": 485, 2024-06-26T07:04:44.6009160Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:04:44.6009820Z }, 2024-06-26T07:04:44.6010071Z { 2024-06-26T07:04:44.6010502Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6011157Z "size": 91710937, 2024-06-26T07:04:44.6011730Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:04:44.6012357Z }, 2024-06-26T07:04:44.6012607Z { 2024-06-26T07:04:44.6013033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6013695Z "size": 3233, 2024-06-26T07:04:44.6014250Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:04:44.6014878Z }, 2024-06-26T07:04:44.6015123Z { 2024-06-26T07:04:44.6015546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6016094Z "size": 1905, 2024-06-26T07:04:44.6016651Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:04:44.6017266Z }, 2024-06-26T07:04:44.6017513Z { 2024-06-26T07:04:44.6017949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6018509Z "size": 702, 2024-06-26T07:04:44.6019048Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:44.6019671Z }, 2024-06-26T07:04:44.6019927Z { 2024-06-26T07:04:44.6020357Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6020958Z "size": 2784789002, 2024-06-26T07:04:44.6021536Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:04:44.6022139Z }, 2024-06-26T07:04:44.6022378Z { 2024-06-26T07:04:44.6022815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6023365Z "size": 381, 2024-06-26T07:04:44.6023905Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:04:44.6024788Z }, 2024-06-26T07:04:44.6025041Z { 2024-06-26T07:04:44.6025478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6026034Z "size": 11879, 2024-06-26T07:04:44.6026562Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:04:44.6027178Z }, 2024-06-26T07:04:44.6027428Z { 2024-06-26T07:04:44.6027872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6028420Z "size": 803, 2024-06-26T07:04:44.6028986Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:04:44.6029607Z }, 2024-06-26T07:04:44.6029847Z { 2024-06-26T07:04:44.6030275Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6030831Z "size": 106, 2024-06-26T07:04:44.6031368Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:04:44.6031969Z }, 2024-06-26T07:04:44.6032228Z { 2024-06-26T07:04:44.6032656Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6033211Z "size": 503, 2024-06-26T07:04:44.6033737Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:04:44.6034347Z }, 2024-06-26T07:04:44.6034600Z { 2024-06-26T07:04:44.6035026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6035584Z "size": 121477177, 2024-06-26T07:04:44.6036147Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:04:44.6036779Z }, 2024-06-26T07:04:44.6037024Z { 2024-06-26T07:04:44.6037455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6038006Z "size": 109, 2024-06-26T07:04:44.6038549Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:04:44.6039152Z }, 2024-06-26T07:04:44.6039411Z { 2024-06-26T07:04:44.6039845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6040400Z "size": 491, 2024-06-26T07:04:44.6040934Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:04:44.6041539Z }, 2024-06-26T07:04:44.6041797Z { 2024-06-26T07:04:44.6042220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6042772Z "size": 291, 2024-06-26T07:04:44.6043324Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:04:44.6044042Z }, 2024-06-26T07:04:44.6044289Z { 2024-06-26T07:04:44.6044726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6045274Z "size": 101, 2024-06-26T07:04:44.6045816Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:04:44.6046412Z }, 2024-06-26T07:04:44.6046663Z { 2024-06-26T07:04:44.6047097Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6047644Z "size": 1473, 2024-06-26T07:04:44.6048185Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:04:44.6048803Z }, 2024-06-26T07:04:44.6049062Z { 2024-06-26T07:04:44.6049484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6050095Z "size": 446710781, 2024-06-26T07:04:44.6050656Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:04:44.6051424Z }, 2024-06-26T07:04:44.6051676Z { 2024-06-26T07:04:44.6052120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6052670Z "size": 161, 2024-06-26T07:04:44.6053194Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:04:44.6053784Z }, 2024-06-26T07:04:44.6054033Z { 2024-06-26T07:04:44.6054465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6055018Z "size": 564, 2024-06-26T07:04:44.6055564Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:04:44.6056191Z }, 2024-06-26T07:04:44.6056445Z { 2024-06-26T07:04:44.6056878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6057439Z "size": 35874600, 2024-06-26T07:04:44.6058003Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:04:44.6058622Z }, 2024-06-26T07:04:44.6058866Z { 2024-06-26T07:04:44.6059314Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6059916Z "size": 104, 2024-06-26T07:04:44.6060439Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:04:44.6061042Z }, 2024-06-26T07:04:44.6061295Z { 2024-06-26T07:04:44.6061734Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6062280Z "size": 425, 2024-06-26T07:04:44.6062814Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:04:44.6063423Z }, 2024-06-26T07:04:44.6063677Z { 2024-06-26T07:04:44.6064100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6064835Z "size": 20262084, 2024-06-26T07:04:44.6065394Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:04:44.6066014Z }, 2024-06-26T07:04:44.6066254Z { 2024-06-26T07:04:44.6066701Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6067250Z "size": 439, 2024-06-26T07:04:44.6067770Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:04:44.6068369Z }, 2024-06-26T07:04:44.6068621Z { 2024-06-26T07:04:44.6069055Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6069612Z "size": 702, 2024-06-26T07:04:44.6070207Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:44.6070847Z }, 2024-06-26T07:04:44.6071102Z { 2024-06-26T07:04:44.6071529Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6072102Z "size": 143, 2024-06-26T07:04:44.6072636Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:04:44.6073245Z }, 2024-06-26T07:04:44.6073493Z { 2024-06-26T07:04:44.6073933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6074594Z "size": 135, 2024-06-26T07:04:44.6075142Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:04:44.6075766Z }, 2024-06-26T07:04:44.6076022Z { 2024-06-26T07:04:44.6076463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6077018Z "size": 4911495247, 2024-06-26T07:04:44.6077592Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:04:44.6078206Z }, 2024-06-26T07:04:44.6078456Z { 2024-06-26T07:04:44.6078878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6079431Z "size": 190, 2024-06-26T07:04:44.6079992Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:04:44.6080614Z }, 2024-06-26T07:04:44.6080868Z { 2024-06-26T07:04:44.6081306Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6081932Z "size": 565, 2024-06-26T07:04:44.6082465Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:04:44.6083068Z }, 2024-06-26T07:04:44.6083312Z { 2024-06-26T07:04:44.6083739Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6084287Z "size": 43163095, 2024-06-26T07:04:44.6084841Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:04:44.6085463Z }, 2024-06-26T07:04:44.6085701Z { 2024-06-26T07:04:44.6086130Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6086683Z "size": 106, 2024-06-26T07:04:44.6087223Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:04:44.6087835Z }, 2024-06-26T07:04:44.6088083Z { 2024-06-26T07:04:44.6088518Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6089073Z "size": 1213, 2024-06-26T07:04:44.6089662Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:04:44.6090275Z }, 2024-06-26T07:04:44.6090528Z { 2024-06-26T07:04:44.6091025Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6091592Z "size": 702, 2024-06-26T07:04:44.6092149Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:44.6092762Z }, 2024-06-26T07:04:44.6092999Z { 2024-06-26T07:04:44.6093428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6093981Z "size": 138, 2024-06-26T07:04:44.6094543Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:04:44.6095155Z }, 2024-06-26T07:04:44.6095412Z { 2024-06-26T07:04:44.6095845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6096395Z "size": 120, 2024-06-26T07:04:44.6096933Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:04:44.6097563Z }, 2024-06-26T07:04:44.6097809Z { 2024-06-26T07:04:44.6098236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6098778Z "size": 1904157495, 2024-06-26T07:04:44.6099348Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:04:44.6099994Z }, 2024-06-26T07:04:44.6100251Z { 2024-06-26T07:04:44.6100676Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6101223Z "size": 172, 2024-06-26T07:04:44.6101759Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:04:44.6102369Z }, 2024-06-26T07:04:44.6102612Z { 2024-06-26T07:04:44.6103036Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6103584Z "size": 908, 2024-06-26T07:04:44.6104130Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:04:44.6105101Z }, 2024-06-26T07:04:44.6105357Z { 2024-06-26T07:04:44.6105790Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6106346Z "size": 702, 2024-06-26T07:04:44.6106887Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:44.6107502Z }, 2024-06-26T07:04:44.6107738Z { 2024-06-26T07:04:44.6108167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6108723Z "size": 134, 2024-06-26T07:04:44.6109262Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:04:44.6109859Z }, 2024-06-26T07:04:44.6110105Z { 2024-06-26T07:04:44.6110542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6111098Z "size": 32, 2024-06-26T07:04:44.6111638Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:44.6112324Z }, 2024-06-26T07:04:44.6112584Z { 2024-06-26T07:04:44.6113006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6113563Z "size": 156, 2024-06-26T07:04:44.6114110Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:04:44.6114741Z }, 2024-06-26T07:04:44.6114978Z { 2024-06-26T07:04:44.6115418Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6115970Z "size": 1841, 2024-06-26T07:04:44.6116527Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:04:44.6117135Z }, 2024-06-26T07:04:44.6117381Z { 2024-06-26T07:04:44.6117818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6118368Z "size": 7529784, 2024-06-26T07:04:44.6118917Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:04:44.6119540Z }, 2024-06-26T07:04:44.6119792Z { 2024-06-26T07:04:44.6120225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6120778Z "size": 165, 2024-06-26T07:04:44.6121321Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:04:44.6121938Z }, 2024-06-26T07:04:44.6122179Z { 2024-06-26T07:04:44.6122616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6123171Z "size": 7944, 2024-06-26T07:04:44.6123720Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:04:44.6124334Z }, 2024-06-26T07:04:44.6124588Z { 2024-06-26T07:04:44.6125025Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6125572Z "size": 8070, 2024-06-26T07:04:44.6126132Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:04:44.6126758Z }, 2024-06-26T07:04:44.6127014Z { 2024-06-26T07:04:44.6127447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6128014Z "size": 303, 2024-06-26T07:04:44.6128567Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:04:44.6129189Z }, 2024-06-26T07:04:44.6129432Z { 2024-06-26T07:04:44.6129926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6130492Z "size": 7629071, 2024-06-26T07:04:44.6131088Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:04:44.6131689Z }, 2024-06-26T07:04:44.6131951Z { 2024-06-26T07:04:44.6132395Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6132943Z "size": 108, 2024-06-26T07:04:44.6133492Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:04:44.6134115Z }, 2024-06-26T07:04:44.6134378Z { 2024-06-26T07:04:44.6134808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6135444Z "size": 54145775, 2024-06-26T07:04:44.6136004Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:04:44.6136635Z }, 2024-06-26T07:04:44.6136879Z { 2024-06-26T07:04:44.6137319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6137888Z "size": 474, 2024-06-26T07:04:44.6138427Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:04:44.6139042Z }, 2024-06-26T07:04:44.6139293Z { 2024-06-26T07:04:44.6139721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6140277Z "size": 1374858747, 2024-06-26T07:04:44.6140842Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:04:44.6141446Z }, 2024-06-26T07:04:44.6141702Z { 2024-06-26T07:04:44.6142133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6142745Z "size": 106, 2024-06-26T07:04:44.6143311Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:04:44.6143935Z }, 2024-06-26T07:04:44.6144179Z { 2024-06-26T07:04:44.6144779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6145342Z "size": 558, 2024-06-26T07:04:44.6145890Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:04:44.6146521Z }, 2024-06-26T07:04:44.6146772Z { 2024-06-26T07:04:44.6147204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6147754Z "size": 46248513, 2024-06-26T07:04:44.6148313Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:04:44.6148923Z }, 2024-06-26T07:04:44.6149171Z { 2024-06-26T07:04:44.6149631Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6150216Z "size": 111, 2024-06-26T07:04:44.6150777Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:04:44.6151387Z }, 2024-06-26T07:04:44.6151644Z { 2024-06-26T07:04:44.6152070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6152629Z "size": 32, 2024-06-26T07:04:44.6153174Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:44.6153799Z }, 2024-06-26T07:04:44.6154053Z { 2024-06-26T07:04:44.6154493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6155040Z "size": 32, 2024-06-26T07:04:44.6155589Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:44.6156206Z }, 2024-06-26T07:04:44.6156445Z { 2024-06-26T07:04:44.6156878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6157430Z "size": 32, 2024-06-26T07:04:44.6157988Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:44.6158604Z }, 2024-06-26T07:04:44.6158864Z { 2024-06-26T07:04:44.6159312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:44.6159925Z "size": 32, 2024-06-26T07:04:44.6160469Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:44.6161091Z } 2024-06-26T07:04:44.6161347Z ] 2024-06-26T07:04:44.6161597Z } 2024-06-26T07:04:44.6283843Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:44.6284343Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:44.6284903Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:04:44.6293132Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:44.6293635Z env: 2024-06-26T07:04:44.6293915Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:44.6295071Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:44.6296264Z ##[endgroup] 2024-06-26T07:04:44.6319541Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:44.6368648Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:04:44.6369230Z with: 2024-06-26T07:04:44.6370322Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:44.6371759Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:44.6372285Z env: 2024-06-26T07:04:44.6372568Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:44.6372921Z ##[endgroup] 2024-06-26T07:04:44.6388316Z ##[group]Run set -x 2024-06-26T07:04:44.6388667Z set -x 2024-06-26T07:04:44.6388975Z set +e 2024-06-26T07:04:44.6389447Z  2024-06-26T07:04:44.6389739Z login() { 2024-06-26T07:04:44.6390470Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:44.6391202Z } 2024-06-26T07:04:44.6391505Z  2024-06-26T07:04:44.6391832Z retry () { 2024-06-26T07:04:44.6392250Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:44.6392707Z } 2024-06-26T07:04:44.6392994Z  2024-06-26T07:04:44.6393329Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:44.6393748Z  2024-06-26T07:04:44.6394043Z set -e 2024-06-26T07:04:44.6394537Z # ignore output since only exit code is used for conditional 2024-06-26T07:04:44.6395253Z # only pull docker image if it's not available locally 2024-06-26T07:04:44.6396025Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:04:44.6396745Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:04:44.6397217Z fi 2024-06-26T07:04:44.6405039Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:44.6405538Z env: 2024-06-26T07:04:44.6405824Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:44.6406974Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:44.6408282Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:44.6408804Z ##[endgroup] 2024-06-26T07:04:44.6427394Z + set +e 2024-06-26T07:04:44.6427893Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:44.6428520Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:44.6429401Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:44.6431170Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:45.0588346Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:45.0589384Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:45.0590255Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:45.0590794Z 2024-06-26T07:04:45.0590915Z Login Succeeded 2024-06-26T07:04:45.0598360Z + set -e 2024-06-26T07:04:45.0599867Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.0723037Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.0725170Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:45.2838892Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:04:45.2843418Z 7a2c55901189: Pulling fs layer 2024-06-26T07:04:45.2843991Z 224fe954d725: Pulling fs layer 2024-06-26T07:04:45.2844780Z 75722010b82e: Pulling fs layer 2024-06-26T07:04:45.2845337Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:04:45.2845854Z b57676e46aee: Pulling fs layer 2024-06-26T07:04:45.2846414Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:04:45.2846947Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:04:45.2847493Z 0c1227890755: Pulling fs layer 2024-06-26T07:04:45.2847974Z d8d1234baab3: Pulling fs layer 2024-06-26T07:04:45.2848436Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:04:45.2848817Z d527cbbb87e3: Waiting 2024-06-26T07:04:45.2849267Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:04:45.2849778Z 309b872da805: Pulling fs layer 2024-06-26T07:04:45.2850291Z e82932cb002c: Pulling fs layer 2024-06-26T07:04:45.2850781Z b57676e46aee: Waiting 2024-06-26T07:04:45.2851545Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:04:45.2852471Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:04:45.2852954Z a8c1e85b5e14: Waiting 2024-06-26T07:04:45.2853372Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:04:45.2853752Z f4863801e30c: Pulling fs layer 2024-06-26T07:04:45.2854126Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:04:45.2854497Z cf2f34f225db: Pulling fs layer 2024-06-26T07:04:45.2854868Z df546d7b3e34: Pulling fs layer 2024-06-26T07:04:45.2855218Z d8d1234baab3: Waiting 2024-06-26T07:04:45.2855546Z a41a8d1c11c8: Waiting 2024-06-26T07:04:45.2855878Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:04:45.2856243Z 96d0e2032184: Pulling fs layer 2024-06-26T07:04:45.2856610Z 06e12ce59790: Pulling fs layer 2024-06-26T07:04:45.2856982Z ed712f63cde7: Pulling fs layer 2024-06-26T07:04:45.2857357Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:04:45.2857726Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:04:45.2858088Z 7ed32bc8e469: Waiting 2024-06-26T07:04:45.2858413Z 4bdf41c9ecbe: Waiting 2024-06-26T07:04:45.2858735Z ec1e7978c1fe: Waiting 2024-06-26T07:04:45.2859048Z 309b872da805: Waiting 2024-06-26T07:04:45.2859472Z 06e12ce59790: Waiting 2024-06-26T07:04:45.2859789Z d37dd9ed8948: Waiting 2024-06-26T07:04:45.2860205Z 3d914950f8ad: Waiting 2024-06-26T07:04:45.2860541Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:04:45.2861069Z 96d0e2032184: Waiting 2024-06-26T07:04:45.2861396Z de45a14c9b22: Pulling fs layer 2024-06-26T07:04:45.2861750Z e82932cb002c: Waiting 2024-06-26T07:04:45.2862063Z cf2f34f225db: Waiting 2024-06-26T07:04:45.2862379Z a97a71ef8a4e: Waiting 2024-06-26T07:04:45.2862697Z e99241e77209: Pulling fs layer 2024-06-26T07:04:45.2863060Z 25cd0c482ad6: Waiting 2024-06-26T07:04:45.2863394Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:04:45.2863797Z ca9dd73e3e93: Waiting 2024-06-26T07:04:45.2864127Z a823ffa4c541: Pulling fs layer 2024-06-26T07:04:45.2864632Z 9bf94bc45658: Waiting 2024-06-26T07:04:45.2864963Z 44ba272eae20: Pulling fs layer 2024-06-26T07:04:45.2865336Z da02d278e524: Pulling fs layer 2024-06-26T07:04:45.2865700Z df546d7b3e34: Waiting 2024-06-26T07:04:45.2866020Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:04:45.2866476Z a823ffa4c541: Waiting 2024-06-26T07:04:45.2866802Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:04:45.2867230Z ed712f63cde7: Waiting 2024-06-26T07:04:45.2867563Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:04:45.2867926Z 73a0ce681c29: Waiting 2024-06-26T07:04:45.2868256Z b2950a036534: Pulling fs layer 2024-06-26T07:04:45.2868691Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:04:45.2869052Z 0c1227890755: Waiting 2024-06-26T07:04:45.2869377Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:04:45.2869733Z b2950a036534: Waiting 2024-06-26T07:04:45.2870049Z 7a868239227f: Pulling fs layer 2024-06-26T07:04:45.2870468Z 25e19d81d9fc: Waiting 2024-06-26T07:04:45.2870793Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:04:45.2871284Z 7a868239227f: Waiting 2024-06-26T07:04:45.2871609Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:04:45.2871981Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:04:45.2872355Z c1068d03a811: Pulling fs layer 2024-06-26T07:04:45.2872715Z 671b4a301d81: Pulling fs layer 2024-06-26T07:04:45.2873069Z 44ba272eae20: Waiting 2024-06-26T07:04:45.2873399Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:04:45.2873917Z 84b2c7a89982: Waiting 2024-06-26T07:04:45.2874255Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:04:45.2874631Z fb2e97768560: Pulling fs layer 2024-06-26T07:04:45.2874993Z 671b4a301d81: Waiting 2024-06-26T07:04:45.2875296Z 83911d7c7ecf: Waiting 2024-06-26T07:04:45.2875611Z b3c5d66c679f: Waiting 2024-06-26T07:04:45.2875939Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:04:45.2876300Z fb2e97768560: Waiting 2024-06-26T07:04:45.2876614Z e08e88c13668: Pulling fs layer 2024-06-26T07:04:45.2876989Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:04:45.2877349Z de45a14c9b22: Waiting 2024-06-26T07:04:45.2877655Z 9cd4c745f8e4: Waiting 2024-06-26T07:04:45.2878080Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:04:45.2878443Z b599c5a8c7cb: Waiting 2024-06-26T07:04:45.2878905Z e08e88c13668: Waiting 2024-06-26T07:04:45.2879222Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:04:45.2879585Z cf7c2cb896cb: Waiting 2024-06-26T07:04:45.2879906Z ddc9740bcea2: Waiting 2024-06-26T07:04:45.2880237Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:04:45.2880615Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:04:45.2880993Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:04:45.2881354Z 4f4fb700ef54: Waiting 2024-06-26T07:04:45.2881676Z e797fe86caef: Pulling fs layer 2024-06-26T07:04:45.2882039Z 236c8f0245f4: Waiting 2024-06-26T07:04:45.2882365Z 967190076fc0: Pulling fs layer 2024-06-26T07:04:45.2882725Z dda1ce302b9a: Waiting 2024-06-26T07:04:45.2883033Z e797fe86caef: Waiting 2024-06-26T07:04:45.2883362Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:04:45.2883739Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:04:45.2884086Z 967190076fc0: Waiting 2024-06-26T07:04:45.2884413Z cc91933e985a: Pulling fs layer 2024-06-26T07:04:45.2884783Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:04:45.2885160Z 526083994857: Pulling fs layer 2024-06-26T07:04:45.2885535Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:04:45.2885908Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:04:45.2886271Z cc91933e985a: Waiting 2024-06-26T07:04:45.2886589Z 9f3187916e7d: Waiting 2024-06-26T07:04:45.2886919Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:04:45.2887292Z 95237e7311a1: Pulling fs layer 2024-06-26T07:04:45.2887666Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:04:45.2888023Z 05a6f6dcc1c0: Waiting 2024-06-26T07:04:45.2888358Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:04:45.2888727Z 2060028f7488: Pulling fs layer 2024-06-26T07:04:45.2889096Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:04:45.2889450Z 6bc9dc8315b0: Waiting 2024-06-26T07:04:45.2889770Z 95237e7311a1: Waiting 2024-06-26T07:04:45.2890085Z de0f8a8c45c6: Waiting 2024-06-26T07:04:45.2890399Z 2060028f7488: Waiting 2024-06-26T07:04:45.2890695Z 526083994857: Waiting 2024-06-26T07:04:45.4166516Z 224fe954d725: Verifying Checksum 2024-06-26T07:04:45.4166999Z 224fe954d725: Download complete 2024-06-26T07:04:45.5210102Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:04:45.5210560Z d527cbbb87e3: Download complete 2024-06-26T07:04:45.5906258Z b57676e46aee: Download complete 2024-06-26T07:04:45.6273768Z 7a2c55901189: Verifying Checksum 2024-06-26T07:04:45.6274213Z 7a2c55901189: Download complete 2024-06-26T07:04:45.7101751Z a41a8d1c11c8: Download complete 2024-06-26T07:04:45.7851756Z 0c1227890755: Download complete 2024-06-26T07:04:45.8643392Z d8d1234baab3: Verifying Checksum 2024-06-26T07:04:45.8643986Z d8d1234baab3: Download complete 2024-06-26T07:04:45.9132649Z 75722010b82e: Verifying Checksum 2024-06-26T07:04:45.9133087Z 75722010b82e: Download complete 2024-06-26T07:04:46.0009876Z ec1e7978c1fe: Download complete 2024-06-26T07:04:46.0767045Z 309b872da805: Download complete 2024-06-26T07:04:46.6464831Z 7a2c55901189: Pull complete 2024-06-26T07:04:46.9301179Z 224fe954d725: Pull complete 2024-06-26T07:04:47.8634827Z 75722010b82e: Pull complete 2024-06-26T07:04:47.9038605Z d527cbbb87e3: Pull complete 2024-06-26T07:04:47.9532132Z b57676e46aee: Pull complete 2024-06-26T07:04:48.6360826Z e82932cb002c: Verifying Checksum 2024-06-26T07:04:48.6361417Z e82932cb002c: Download complete 2024-06-26T07:04:48.7110192Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:04:48.7110825Z ca9dd73e3e93: Download complete 2024-06-26T07:04:48.7739434Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:04:48.7739936Z 16d76a79d5d6: Download complete 2024-06-26T07:04:48.8521877Z 4bdf41c9ecbe: Download complete 2024-06-26T07:04:49.8416658Z f4863801e30c: Verifying Checksum 2024-06-26T07:04:49.8417150Z f4863801e30c: Download complete 2024-06-26T07:04:49.9072389Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:04:49.9072822Z d37dd9ed8948: Download complete 2024-06-26T07:04:49.9681072Z cf2f34f225db: Verifying Checksum 2024-06-26T07:04:49.9681588Z cf2f34f225db: Download complete 2024-06-26T07:04:50.0340411Z df546d7b3e34: Download complete 2024-06-26T07:04:58.9349042Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:04:58.9349979Z a8c1e85b5e14: Download complete 2024-06-26T07:04:59.0049566Z 96d0e2032184: Verifying Checksum 2024-06-26T07:04:59.0050183Z 96d0e2032184: Download complete 2024-06-26T07:04:59.0719725Z 06e12ce59790: Download complete 2024-06-26T07:04:59.1479483Z ed712f63cde7: Verifying Checksum 2024-06-26T07:04:59.1480098Z ed712f63cde7: Download complete 2024-06-26T07:04:59.2258916Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:04:59.2259435Z a97a71ef8a4e: Download complete 2024-06-26T07:04:59.2928088Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:04:59.2928551Z 25cd0c482ad6: Download complete 2024-06-26T07:05:00.5609765Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:05:00.5610448Z 9bf94bc45658: Download complete 2024-06-26T07:05:00.6338864Z de45a14c9b22: Verifying Checksum 2024-06-26T07:05:00.6339355Z de45a14c9b22: Download complete 2024-06-26T07:05:00.7099101Z e99241e77209: Download complete 2024-06-26T07:05:00.7752413Z 35e8f4e2a42f: Download complete 2024-06-26T07:05:00.8398054Z a823ffa4c541: Verifying Checksum 2024-06-26T07:05:00.8398718Z a823ffa4c541: Download complete 2024-06-26T07:05:00.9030090Z 44ba272eae20: Download complete 2024-06-26T07:05:05.4147822Z da02d278e524: Verifying Checksum 2024-06-26T07:05:05.4148652Z da02d278e524: Download complete 2024-06-26T07:05:05.4895365Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:05:05.4895928Z 73a0ce681c29: Download complete 2024-06-26T07:05:05.5753117Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:05:05.5753708Z 49737bd5ae70: Download complete 2024-06-26T07:05:05.9814415Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:05:05.9815000Z 9cd4c745f8e4: Download complete 2024-06-26T07:05:06.1145635Z b2950a036534: Verifying Checksum 2024-06-26T07:05:06.1146233Z b2950a036534: Download complete 2024-06-26T07:05:06.1774170Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:05:06.1774674Z 25e19d81d9fc: Download complete 2024-06-26T07:05:06.4353525Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:05:06.4354173Z 60156b40fcbe: Download complete 2024-06-26T07:05:06.5078037Z 7a868239227f: Verifying Checksum 2024-06-26T07:05:06.5078576Z 7a868239227f: Download complete 2024-06-26T07:05:06.5762369Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:05:06.5762930Z 83911d7c7ecf: Download complete 2024-06-26T07:05:06.6419853Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:05:06.6420362Z b3c5d66c679f: Download complete 2024-06-26T07:05:11.2090267Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:05:11.2090770Z 7ed32bc8e469: Download complete 2024-06-26T07:05:11.2971207Z c1068d03a811: Download complete 2024-06-26T07:05:11.3666133Z 671b4a301d81: Download complete 2024-06-26T07:05:11.8429005Z a8c1e85b5e14: Pull complete 2024-06-26T07:05:11.8511783Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:05:11.8512309Z 67760cf5ec46: Download complete 2024-06-26T07:05:11.9364820Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:05:11.9365465Z ddc9740bcea2: Download complete 2024-06-26T07:05:12.0207974Z fb2e97768560: Verifying Checksum 2024-06-26T07:05:12.0208432Z fb2e97768560: Download complete 2024-06-26T07:05:12.0686027Z a41a8d1c11c8: Pull complete 2024-06-26T07:05:12.0939049Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:05:12.0939668Z b599c5a8c7cb: Download complete 2024-06-26T07:05:12.1640612Z e08e88c13668: Verifying Checksum 2024-06-26T07:05:12.1641211Z e08e88c13668: Download complete 2024-06-26T07:05:12.2713708Z 0c1227890755: Pull complete 2024-06-26T07:05:12.4797321Z d8d1234baab3: Pull complete 2024-06-26T07:05:17.9284486Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:05:17.9285073Z 3d914950f8ad: Download complete 2024-06-26T07:05:18.0031169Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:05:18.0031866Z cf7c2cb896cb: Download complete 2024-06-26T07:05:18.0732697Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:05:18.0733364Z 3a778ddfac2a: Download complete 2024-06-26T07:05:18.1599779Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:05:18.1600249Z 236c8f0245f4: Download complete 2024-06-26T07:05:18.1694757Z 4f4fb700ef54: Download complete 2024-06-26T07:05:18.2357642Z dda1ce302b9a: Download complete 2024-06-26T07:05:18.3133338Z e797fe86caef: Download complete 2024-06-26T07:05:18.4388258Z 967190076fc0: Verifying Checksum 2024-06-26T07:05:18.4388737Z 967190076fc0: Download complete 2024-06-26T07:05:18.4959725Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:05:18.4960479Z 4fb0a83f0685: Download complete 2024-06-26T07:05:18.5641500Z 68af2ec4e30a: Verifying Checksum 2024-06-26T07:05:18.5641990Z 68af2ec4e30a: Download complete 2024-06-26T07:05:18.6317506Z cc91933e985a: Download complete 2024-06-26T07:05:18.6975084Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:05:18.6975827Z 9f3187916e7d: Download complete 2024-06-26T07:05:18.8247825Z 526083994857: Verifying Checksum 2024-06-26T07:05:18.8248343Z 526083994857: Download complete 2024-06-26T07:05:18.9067147Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:05:18.9067740Z 05a6f6dcc1c0: Download complete 2024-06-26T07:05:19.5145395Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:05:19.5145965Z 6bc9dc8315b0: Download complete 2024-06-26T07:05:19.5990233Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:05:19.5990722Z 1b0e27c35f07: Download complete 2024-06-26T07:05:41.5364540Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:05:41.5365142Z 12ba7431ff38: Download complete 2024-06-26T07:05:41.6120031Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:05:41.6120652Z 789eff6c51aa: Download complete 2024-06-26T07:05:41.6903004Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:05:41.6903815Z de0f8a8c45c6: Download complete 2024-06-26T07:05:43.0030061Z 2060028f7488: Verifying Checksum 2024-06-26T07:05:43.0030520Z 2060028f7488: Download complete 2024-06-26T07:05:43.0841882Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:05:43.0842318Z e412c7f3b2de: Download complete 2024-06-26T07:05:47.4561034Z 95237e7311a1: Verifying Checksum 2024-06-26T07:05:47.4561528Z 95237e7311a1: Download complete 2024-06-26T07:06:09.9934920Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:06:09.9935381Z 84b2c7a89982: Download complete 2024-06-26T07:06:28.1100713Z 7ed32bc8e469: Pull complete 2024-06-26T07:06:28.3465031Z ec1e7978c1fe: Pull complete 2024-06-26T07:06:28.5450235Z 309b872da805: Pull complete 2024-06-26T07:06:35.8178115Z e82932cb002c: Pull complete 2024-06-26T07:06:36.0262832Z ca9dd73e3e93: Pull complete 2024-06-26T07:06:36.2268114Z 16d76a79d5d6: Pull complete 2024-06-26T07:06:36.4513450Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:06:38.8589008Z f4863801e30c: Pull complete 2024-06-26T07:06:39.0715105Z d37dd9ed8948: Pull complete 2024-06-26T07:06:39.2742580Z cf2f34f225db: Pull complete 2024-06-26T07:06:39.4770531Z df546d7b3e34: Pull complete 2024-06-26T07:07:28.3491541Z 3d914950f8ad: Pull complete 2024-06-26T07:07:28.4779128Z 96d0e2032184: Pull complete 2024-06-26T07:07:28.6267540Z 06e12ce59790: Pull complete 2024-06-26T07:07:28.8556783Z ed712f63cde7: Pull complete 2024-06-26T07:07:29.0337298Z a97a71ef8a4e: Pull complete 2024-06-26T07:07:29.1856683Z 25cd0c482ad6: Pull complete 2024-06-26T07:07:32.7104710Z 9bf94bc45658: Pull complete 2024-06-26T07:07:32.8607912Z de45a14c9b22: Pull complete 2024-06-26T07:07:32.9732632Z e99241e77209: Pull complete 2024-06-26T07:07:33.0751198Z 35e8f4e2a42f: Pull complete 2024-06-26T07:07:33.1831301Z a823ffa4c541: Pull complete 2024-06-26T07:07:33.3005464Z 44ba272eae20: Pull complete 2024-06-26T07:07:41.6267492Z da02d278e524: Pull complete 2024-06-26T07:07:41.8088368Z 73a0ce681c29: Pull complete 2024-06-26T07:07:41.9761857Z 49737bd5ae70: Pull complete 2024-06-26T07:07:42.9392265Z 9cd4c745f8e4: Pull complete 2024-06-26T07:07:43.1358200Z b2950a036534: Pull complete 2024-06-26T07:07:43.3137548Z 25e19d81d9fc: Pull complete 2024-06-26T07:07:43.6805191Z 60156b40fcbe: Pull complete 2024-06-26T07:07:43.8640609Z 7a868239227f: Pull complete 2024-06-26T07:07:44.2724451Z 83911d7c7ecf: Pull complete 2024-06-26T07:07:44.4189476Z b3c5d66c679f: Pull complete 2024-06-26T07:08:23.0113083Z 84b2c7a89982: Pull complete 2024-06-26T07:08:23.2084034Z c1068d03a811: Pull complete 2024-06-26T07:08:23.4089387Z 671b4a301d81: Pull complete 2024-06-26T07:08:25.4716416Z 67760cf5ec46: Pull complete 2024-06-26T07:08:25.5274668Z ddc9740bcea2: Pull complete 2024-06-26T07:08:25.5782436Z fb2e97768560: Pull complete 2024-06-26T07:08:25.6710788Z b599c5a8c7cb: Pull complete 2024-06-26T07:08:25.7057514Z e08e88c13668: Pull complete 2024-06-26T07:08:59.7152528Z 12ba7431ff38: Pull complete 2024-06-26T07:08:59.9102475Z cf7c2cb896cb: Pull complete 2024-06-26T07:09:00.1092114Z 3a778ddfac2a: Pull complete 2024-06-26T07:09:00.5993215Z 236c8f0245f4: Pull complete 2024-06-26T07:09:00.8018468Z 4f4fb700ef54: Pull complete 2024-06-26T07:09:01.0295860Z dda1ce302b9a: Pull complete 2024-06-26T07:09:01.2553811Z e797fe86caef: Pull complete 2024-06-26T07:09:01.5961886Z 967190076fc0: Pull complete 2024-06-26T07:09:01.8211415Z 4fb0a83f0685: Pull complete 2024-06-26T07:09:01.9518945Z 68af2ec4e30a: Pull complete 2024-06-26T07:09:02.1493091Z cc91933e985a: Pull complete 2024-06-26T07:09:02.3471159Z 9f3187916e7d: Pull complete 2024-06-26T07:09:03.4128599Z 526083994857: Pull complete 2024-06-26T07:09:03.6195252Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:09:05.6941161Z 6bc9dc8315b0: Pull complete 2024-06-26T07:09:05.8912638Z 1b0e27c35f07: Pull complete 2024-06-26T07:09:19.3871668Z 95237e7311a1: Pull complete 2024-06-26T07:09:19.5022050Z 789eff6c51aa: Pull complete 2024-06-26T07:09:19.5469115Z de0f8a8c45c6: Pull complete 2024-06-26T07:09:20.2449951Z 2060028f7488: Pull complete 2024-06-26T07:09:20.3120431Z e412c7f3b2de: Pull complete 2024-06-26T07:09:20.5717905Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:09:20.5748281Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:20.5774839Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:20.5814834Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:20.5815772Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:20.5824254Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:20.5825173Z env: 2024-06-26T07:09:20.5825514Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:20.5825860Z ##[endgroup] 2024-06-26T07:09:20.5980871Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:09:20.5981418Z with: 2024-06-26T07:09:20.5981709Z driver-version: 550.54.15 2024-06-26T07:09:20.5982064Z env: 2024-06-26T07:09:20.5982383Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:20.5982730Z ##[endgroup] 2024-06-26T07:09:20.6021645Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:09:20.6022203Z with: 2024-06-26T07:09:20.6022481Z timeout_minutes: 10 2024-06-26T07:09:20.6022804Z max_attempts: 3 2024-06-26T07:09:20.6054357Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:09:20.6086017Z retry_wait_seconds: 10 2024-06-26T07:09:20.6086381Z polling_interval_seconds: 1 2024-06-26T07:09:20.6086754Z warning_on_retry: true 2024-06-26T07:09:20.6087111Z continue_on_error: false 2024-06-26T07:09:20.6087449Z env: 2024-06-26T07:09:20.6087721Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:20.6088076Z DRIVER_VERSION: 550.54.15 2024-06-26T07:09:20.6088427Z ##[endgroup] 2024-06-26T07:09:20.6616949Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:09:20.6618487Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:09:20.6619006Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:09:20.9657778Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:21.0051692Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:09:21.0298799Z No Packages marked for removal 2024-06-26T07:09:21.0417191Z + install_nvidia_driver_common 2024-06-26T07:09:21.0419564Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:09:21.0420215Z Before installing NVIDIA driver 2024-06-26T07:09:21.0422204Z + lspci 2024-06-26T07:09:21.0498859Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:21.0499725Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:21.0500835Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:21.0501893Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:21.0502822Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:21.0503617Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:21.0504334Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:21.0505510Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:21.0506105Z + lsmod 2024-06-26T07:09:21.0513414Z Module Size Used by 2024-06-26T07:09:21.0514047Z xt_nat 16384 0 2024-06-26T07:09:21.0514528Z ib_core 266240 0 2024-06-26T07:09:21.0515052Z nvidia_modeset 1339392 0 2024-06-26T07:09:21.0515548Z veth 16384 0 2024-06-26T07:09:21.0516016Z nvidia_uvm 4599808 0 2024-06-26T07:09:21.0516703Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:21.0517397Z drm 421888 1 nvidia 2024-06-26T07:09:21.0517831Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:21.0518324Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:21.0518825Z xt_conntrack 16384 1 2024-06-26T07:09:21.0519334Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:21.0519796Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:21.0520281Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:21.0520742Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:21.0521302Z xfrm_user 45056 1 2024-06-26T07:09:21.0521785Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:21.0522317Z iptable_nat 16384 1 2024-06-26T07:09:21.0522744Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:21.0523188Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:21.0523675Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:21.0524240Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:21.0525238Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:21.0526105Z xt_addrtype 16384 2 2024-06-26T07:09:21.0526486Z iptable_filter 16384 1 2024-06-26T07:09:21.0526869Z br_netfilter 24576 0 2024-06-26T07:09:21.0527269Z bridge 172032 1 br_netfilter 2024-06-26T07:09:21.0527713Z stp 16384 1 bridge 2024-06-26T07:09:21.0528182Z llc 16384 2 bridge,stp 2024-06-26T07:09:21.0528753Z overlay 86016 0 2024-06-26T07:09:21.0529419Z sunrpc 393216 1 2024-06-26T07:09:21.0529847Z dm_mirror 28672 0 2024-06-26T07:09:21.0530280Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:21.0530771Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:21.0531492Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:21.0532028Z dax 69632 1 dm_mod 2024-06-26T07:09:21.0532432Z crc32_pclmul 16384 0 2024-06-26T07:09:21.0532819Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:21.0533238Z pcbc 16384 0 2024-06-26T07:09:21.0533597Z aesni_intel 188416 0 2024-06-26T07:09:21.0533993Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:21.0534441Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:21.0534864Z mousedev 24576 0 2024-06-26T07:09:21.0535270Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:21.0535829Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:21.0536369Z evdev 20480 3 2024-06-26T07:09:21.0536732Z psmouse 32768 0 2024-06-26T07:09:21.0537099Z button 16384 0 2024-06-26T07:09:21.0537467Z ena 139264 0 2024-06-26T07:09:21.0537837Z ptp 20480 1 ena 2024-06-26T07:09:21.0538227Z pps_core 20480 1 ptp 2024-06-26T07:09:21.0538720Z crc32c_intel 24576 0 2024-06-26T07:09:21.0539102Z autofs4 49152 2 2024-06-26T07:09:21.0539587Z + modinfo nvidia 2024-06-26T07:09:21.0541066Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:21.0542127Z alias: char-major-195-* 2024-06-26T07:09:21.0542690Z version: 550.54.15 2024-06-26T07:09:21.0543218Z supported: external 2024-06-26T07:09:21.0543666Z license: NVIDIA 2024-06-26T07:09:21.0544171Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:21.0545066Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:21.0545537Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:21.0545993Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:21.0546491Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:21.0547161Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:21.0547666Z depends: i2c-core,drm 2024-06-26T07:09:21.0548036Z retpoline: Y 2024-06-26T07:09:21.0548340Z name: nvidia 2024-06-26T07:09:21.0548890Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:21.0549533Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:21.0550171Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:21.0550759Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:21.0551195Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:21.0551615Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:21.0552078Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:21.0552511Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:21.0552948Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:21.0553464Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:21.0554035Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:21.0554510Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:21.0554934Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:21.0555337Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:21.0555788Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:21.0556308Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:21.0556883Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:21.0557445Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:21.0558050Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:21.0558767Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:21.0559372Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:21.0559962Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:21.0560451Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:21.0560988Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:21.0561527Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:21.0562018Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:21.0562476Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:21.0563005Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:21.0563474Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:21.0563911Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:21.0564404Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:21.0564922Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:21.0565395Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:21.0565892Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:21.0566387Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:21.0566869Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:21.0567356Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:21.0567828Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:21.0568235Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:21.0568700Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:21.0569157Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:21.0569607Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:21.0570085Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:21.0570549Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:21.0570993Z parm: rm_firmware_active:charp 2024-06-26T07:09:21.0571536Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:09:21.0571912Z ++ command -v nvidia-smi 2024-06-26T07:09:21.0572304Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:09:21.0572670Z + set +e 2024-06-26T07:09:21.0573251Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:09:23.0540703Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:09:23.0541293Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:23.0542105Z + '[' 0 -ne 0 ']' 2024-06-26T07:09:23.0542489Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:09:23.0543029Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:09:23.0543941Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:09:23.0545135Z + set -e 2024-06-26T07:09:23.0545441Z + '[' 1 -eq 0 ']' 2024-06-26T07:09:23.0546026Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:09:23.0546711Z + post_install_nvidia_driver_common 2024-06-26T07:09:23.0547116Z + sudo modprobe nvidia 2024-06-26T07:09:23.0644974Z + echo 'After installing NVIDIA driver' 2024-06-26T07:09:23.0645662Z + lspci 2024-06-26T07:09:23.0646994Z After installing NVIDIA driver 2024-06-26T07:09:23.0727722Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:23.0729517Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:23.0731496Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:23.0733082Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:23.0734394Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:23.0735149Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:23.0735870Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:23.0736652Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:23.0737243Z + lsmod 2024-06-26T07:09:23.0743302Z Module Size Used by 2024-06-26T07:09:23.0743876Z xt_nat 16384 0 2024-06-26T07:09:23.0745148Z ib_core 266240 0 2024-06-26T07:09:23.0745658Z nvidia_modeset 1339392 0 2024-06-26T07:09:23.0746170Z veth 16384 0 2024-06-26T07:09:23.0746653Z nvidia_uvm 4599808 0 2024-06-26T07:09:23.0747104Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:23.0747595Z drm 421888 1 nvidia 2024-06-26T07:09:23.0748019Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:23.0748485Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:23.0748923Z xt_conntrack 16384 1 2024-06-26T07:09:23.0749304Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:23.0749743Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:23.0750210Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:23.0750645Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:23.0751190Z xfrm_user 45056 1 2024-06-26T07:09:23.0751700Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:23.0752305Z iptable_nat 16384 1 2024-06-26T07:09:23.0752825Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:23.0753358Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:23.0753838Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:23.0754477Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:09:23.0755759Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:09:23.0756627Z xt_addrtype 16384 2 2024-06-26T07:09:23.0757012Z iptable_filter 16384 1 2024-06-26T07:09:23.0757387Z br_netfilter 24576 0 2024-06-26T07:09:23.0757786Z bridge 172032 1 br_netfilter 2024-06-26T07:09:23.0758235Z stp 16384 1 bridge 2024-06-26T07:09:23.0758660Z llc 16384 2 bridge,stp 2024-06-26T07:09:23.0759084Z overlay 86016 0 2024-06-26T07:09:23.0759463Z sunrpc 393216 1 2024-06-26T07:09:23.0760086Z dm_mirror 28672 0 2024-06-26T07:09:23.0760483Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:23.0761037Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:23.0761682Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:23.0762142Z dax 69632 1 dm_mod 2024-06-26T07:09:23.0762547Z crc32_pclmul 16384 0 2024-06-26T07:09:23.0762921Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:23.0763297Z pcbc 16384 0 2024-06-26T07:09:23.0763664Z aesni_intel 188416 0 2024-06-26T07:09:23.0764082Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:23.0764548Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:23.0764967Z mousedev 24576 0 2024-06-26T07:09:23.0765371Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:23.0765917Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:23.0766467Z evdev 20480 3 2024-06-26T07:09:23.0766835Z psmouse 32768 0 2024-06-26T07:09:23.0767202Z button 16384 0 2024-06-26T07:09:23.0767555Z ena 139264 0 2024-06-26T07:09:23.0767932Z ptp 20480 1 ena 2024-06-26T07:09:23.0768329Z pps_core 20480 1 ptp 2024-06-26T07:09:23.0768726Z crc32c_intel 24576 0 2024-06-26T07:09:23.0769092Z autofs4 49152 2 2024-06-26T07:09:23.0769449Z + modinfo nvidia 2024-06-26T07:09:23.0770135Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:23.0770830Z alias: char-major-195-* 2024-06-26T07:09:23.0771294Z version: 550.54.15 2024-06-26T07:09:23.0771640Z supported: external 2024-06-26T07:09:23.0771978Z license: NVIDIA 2024-06-26T07:09:23.0772353Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:23.0772923Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:23.0773380Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:23.0773842Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:23.0774336Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:23.0774829Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:23.0775317Z depends: i2c-core,drm 2024-06-26T07:09:23.0775688Z retpoline: Y 2024-06-26T07:09:23.0775986Z name: nvidia 2024-06-26T07:09:23.0776535Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:23.0777190Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:23.0777839Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:23.0778429Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:23.0778876Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:23.0779314Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:23.0779779Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:23.0780215Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:23.0780657Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:23.0781186Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:23.0781755Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:23.0782229Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:23.0782657Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:23.0783083Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:23.0783542Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:23.0784067Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:23.0784997Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:23.0785561Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:23.0786173Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:23.0786770Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:23.0787385Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:23.0787975Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:23.0788469Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:23.0789090Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:23.0789641Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:23.0790136Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:23.0790598Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:23.0791082Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:23.0791551Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:23.0791997Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:23.0792500Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:23.0793027Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:23.0793503Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:23.0794047Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:23.0794543Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:23.0795032Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:23.0795542Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:23.0796015Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:23.0796419Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:23.0796886Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:23.0797358Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:23.0797820Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:23.0798294Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:23.0798768Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:23.0799223Z parm: rm_firmware_active:charp 2024-06-26T07:09:23.0799618Z + set +e 2024-06-26T07:09:23.0799919Z + nvidia-smi 2024-06-26T07:09:24.6518810Z Wed Jun 26 07:09:24 2024 2024-06-26T07:09:24.6539390Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:24.6540455Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:09:24.6541524Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:24.6542549Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:09:24.6543715Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:09:24.6544986Z | | | MIG M. | 2024-06-26T07:09:24.6545766Z |=========================================+========================+======================| 2024-06-26T07:09:24.6624745Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:09:24.6625770Z | 0% 32C P0 59W / 300W | 0MiB / 23028MiB | 5% Default | 2024-06-26T07:09:24.6626601Z | | | N/A | 2024-06-26T07:09:24.6627499Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:24.6628342Z 2024-06-26T07:09:24.6629238Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:24.6630043Z | Processes: | 2024-06-26T07:09:24.6630953Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:09:24.6631863Z | ID ID Usage | 2024-06-26T07:09:24.6632588Z |=========================================================================================| 2024-06-26T07:09:24.6633434Z | No running processes found | 2024-06-26T07:09:24.6634412Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:25.2675701Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:09:26.8395773Z NVIDIA A10G 2024-06-26T07:09:27.2826548Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:27.2827194Z + '[' 0 -eq 0 ']' 2024-06-26T07:09:27.2827600Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:09:27.2828022Z + set -e 2024-06-26T07:09:27.2828322Z INFO: Ignoring allowed status 0 2024-06-26T07:09:27.2830848Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:09:27.2832569Z + sudo yum install -y yum-utils 2024-06-26T07:09:27.5913056Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:29.0860013Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:09:29.0860752Z Nothing to do 2024-06-26T07:09:29.2352654Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:09:29.2353674Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:29.2354820Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:29.5651473Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:29.5926974Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:29.5928237Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:29.5929188Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:29.6047318Z + sudo yum install -y nvidia-docker2 2024-06-26T07:09:29.9036842Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:31.3652801Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:09:31.3653686Z Nothing to do 2024-06-26T07:09:31.5121977Z + sudo systemctl restart docker 2024-06-26T07:10:12.7117086Z Command completed after 1 attempt(s). 2024-06-26T07:10:12.7179665Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:12.7180439Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:12.7181132Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:10:12.7181794Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:12.7190061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:12.7190548Z env: 2024-06-26T07:10:12.7190822Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:12.7191271Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:12.7191749Z ##[endgroup] 2024-06-26T07:10:12.9401569Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:12.9558331Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:10:12.9658991Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:10:13.0599547Z Prepare all required actions 2024-06-26T07:10:13.0600031Z Getting action download info 2024-06-26T07:10:13.1854488Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:10:13.3484218Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:10:14.5633879Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:10:14.5634368Z with: 2024-06-26T07:10:14.5634703Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:14.5635169Z s3-bucket: gha-artifacts 2024-06-26T07:10:14.5635515Z env: 2024-06-26T07:10:14.5635790Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:14.5636249Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:14.5636754Z ##[endgroup] 2024-06-26T07:10:14.5668560Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:14.5668998Z with: 2024-06-26T07:10:14.5669366Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:14.5669836Z s3-bucket: gha-artifacts 2024-06-26T07:10:14.5670181Z region: us-east-1 2024-06-26T07:10:14.5670485Z env: 2024-06-26T07:10:14.5670760Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:14.5671206Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:14.5671697Z ##[endgroup] 2024-06-26T07:10:14.9791055Z (node:21922) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:14.9791780Z 2024-06-26T07:10:14.9792042Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:14.9792774Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:14.9793646Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:15.0569693Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:10:15.0570799Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:22.5759151Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:22.5765369Z Artifact download has finished successfully 2024-06-26T07:10:22.5922903Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:10:22.5923353Z unzip -o artifacts.zip 2024-06-26T07:10:22.5931136Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:22.5931637Z env: 2024-06-26T07:10:22.5931909Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:22.5932365Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:22.5932849Z ##[endgroup] 2024-06-26T07:10:22.5968403Z Archive: artifacts.zip 2024-06-26T07:10:22.5969873Z creating: dist/ 2024-06-26T07:10:24.5747765Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:24.5748616Z creating: build/custom_test_artifacts/ 2024-06-26T07:10:24.5749269Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:10:24.5750007Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:10:24.5750839Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:24.5754823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:24.5755815Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:24.5756757Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:24.5757763Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:24.5758739Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:24.5759860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:24.5761168Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:24.5762294Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:24.5763419Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:24.5764728Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:24.5766050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:24.5767344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:24.5768627Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:24.5769950Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:24.5771331Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:24.5772366Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:24.5773369Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:24.5809856Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:24.5847606Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:24.5849039Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:24.5892128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:24.5893541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:24.5894961Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:24.5896420Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:24.5897820Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:24.5899200Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:24.5900586Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:24.5902125Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:24.5903469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:24.5905146Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:24.5906408Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:24.5907628Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:24.5908856Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:24.5910039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:24.5911262Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:24.5962293Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:24.6020352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:24.6021535Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:24.6022510Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:24.6023371Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:24.6024285Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:24.6025443Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:10:24.6026469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:24.6027627Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:10:24.6028728Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:10:24.6029768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:10:24.6030850Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:24.6031915Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:10:24.6032999Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:24.6034079Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:10:24.6035196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:10:24.6048398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:10:24.6176176Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:10:24.6177267Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:10:24.6178351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:24.6179551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:10:24.6180710Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:10:24.6181783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:10:24.6183026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:24.6184172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:10:24.6185526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:24.6186658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:10:24.6187778Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:10:24.6201978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:10:24.6281427Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:10:24.6282623Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:24.6283714Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:24.6284812Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:10:24.6285754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:10:24.6286647Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:24.6287546Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:10:24.6288385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:10:24.6289166Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:10:24.6289973Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:10:24.6397726Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:10:24.6458182Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:10:24.6459008Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:10:24.6459720Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:10:24.6460535Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:24.6465865Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:24.6466838Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:24.6467755Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:24.6468727Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:24.6469683Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:24.6470874Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:24.6471970Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:24.6472961Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:24.6473946Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:24.6475089Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:24.6476232Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:24.6477347Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:24.6478470Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:24.6479758Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:24.6480909Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:24.6481926Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:24.6482907Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:24.6519433Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:24.6557204Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:24.6558640Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:24.6601748Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:24.6603169Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:24.6604703Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:24.6606169Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:24.6607560Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:24.6608927Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:24.6610301Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:24.6611763Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:24.6613117Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:24.6614388Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:24.6615669Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:24.6616878Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:24.6618089Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:24.6619273Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:24.6620488Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:24.6671670Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:24.6748684Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:24.6750267Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:24.6751268Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:24.6752117Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:24.6752998Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:24.6753928Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:10:24.6755027Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:10:24.6756362Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:10:24.6757512Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:10:24.6758586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:10:24.6759689Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:10:24.6760798Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:10:24.6761903Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:10:24.6762996Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:10:24.6764076Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:10:24.6765289Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:10:24.6815792Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:10:24.6816970Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:24.6818031Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:24.6818988Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:10:24.6819889Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:10:24.6820751Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:24.6821628Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:10:24.6822471Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:10:24.6823234Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:10:24.6824021Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:10:24.6871543Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:10:24.6872288Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:10:24.6873042Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:10:24.6873918Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:24.6879187Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:24.6880158Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:24.6881137Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:24.6882178Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:24.6883197Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:24.6884339Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:24.6885543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:24.6886586Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:24.6887631Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:24.6888825Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:24.6890154Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:24.6891394Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:24.6892587Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:24.6893802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:24.6895007Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:24.6896108Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:24.6897168Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:24.6933482Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:24.6971395Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:24.6972992Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:24.7015784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:24.7017253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:24.7018730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:24.7020246Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:24.7021723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:24.7035490Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:24.7036956Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:24.7038399Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:24.7039806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:24.7041147Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:24.7042437Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:24.7043711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:24.7044995Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:24.7046245Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:24.7047517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:24.7099703Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:24.7157822Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:24.7159247Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:24.7160420Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:24.7161338Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:24.7162301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:24.7163299Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:10:24.7164421Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:24.7165726Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:10:24.7166939Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:10:24.7168077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:10:24.7169251Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:24.7170509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:10:24.7171793Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:24.7172983Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:10:24.7174145Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:10:24.7175390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:10:24.7289642Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:10:24.7292331Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:10:24.7294691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:24.7296209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:10:24.7297471Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:10:24.7298667Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:10:24.7299892Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:24.7301128Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:10:24.7302360Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:24.7303605Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:10:24.7305110Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:10:24.7312805Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:10:24.7366363Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:10:24.7368965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:24.7371342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:24.7373412Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:10:24.7375204Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:10:24.7376178Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:24.7377129Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:10:24.7378037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:10:24.7378874Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:10:24.7379728Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:10:24.7472531Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:10:24.7513966Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:10:24.7514602Z creating: build/lib/ 2024-06-26T07:10:24.7600104Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:10:24.8039685Z inflating: build/lib/libprotobuf.a 2024-06-26T07:10:24.8049155Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:10:24.8057214Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:10:24.8064966Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:10:24.8065831Z inflating: build/lib/libclog.a 2024-06-26T07:10:24.8083591Z inflating: build/lib/libnnpack.a 2024-06-26T07:10:24.8085795Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:10:24.8148627Z inflating: build/lib/libgtest.a 2024-06-26T07:10:24.8220739Z inflating: build/lib/libbenchmark.a 2024-06-26T07:10:24.8282580Z inflating: build/lib/libasmjit.a 2024-06-26T07:10:24.8290297Z inflating: build/lib/libittnotify.a 2024-06-26T07:10:24.8316860Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:10:24.8439294Z inflating: build/lib/libgloo.a 2024-06-26T07:10:24.8460329Z inflating: build/lib/libfmt.a 2024-06-26T07:10:24.8549710Z inflating: build/lib/libc10.so 2024-06-26T07:10:24.8551025Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:10:24.8552512Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:10:24.8553642Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:10:24.8571906Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:10:24.8589291Z inflating: build/lib/libgmock.a 2024-06-26T07:10:24.8589741Z inflating: build/lib/libgtest_main.a 2024-06-26T07:10:24.8590214Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:10:24.9076247Z inflating: build/lib/libprotoc.a 2024-06-26T07:10:24.9446287Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:10:25.9186137Z inflating: build/lib/libdnnl.a 2024-06-26T07:10:25.9735901Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:10:25.9788832Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:10:25.9789750Z inflating: build/lib/libgmock_main.a 2024-06-26T07:10:26.0992791Z inflating: build/lib/libfbgemm.a 2024-06-26T07:10:26.1237488Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:10:26.1735762Z inflating: build/lib/libkineto.a 2024-06-26T07:10:26.1913527Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:10:26.1954155Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:10:26.2619280Z inflating: build/lib/libonnx.a 2024-06-26T07:10:28.5917000Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:10:28.5921243Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:10:28.5925065Z inflating: build/lib/libshm.so 2024-06-26T07:10:30.5832889Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:10:30.5833675Z inflating: build/lib/libtorch.so 2024-06-26T07:10:30.5836240Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:10:31.3801467Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:10:31.5700079Z inflating: build/lib/libtorch_python.so 2024-06-26T07:10:31.5766752Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:10:31.5786696Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:10:31.5811239Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:10:31.5844939Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:10:31.5845553Z creating: build/bin/ 2024-06-26T07:10:31.5894506Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:10:31.5944338Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:10:31.5993971Z inflating: build/bin/c10_Device_test 2024-06-26T07:10:31.6051446Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:10:31.6103358Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:10:31.6151076Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:10:31.6200615Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:10:31.6253732Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:10:31.6307900Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:10:31.6362714Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:10:31.6431497Z inflating: build/bin/c10_cow_test 2024-06-26T07:10:31.6483102Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:10:31.6531279Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:10:31.6579453Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:10:31.6628729Z inflating: build/bin/c10_Half_test 2024-06-26T07:10:31.6683903Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:10:31.6738059Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:10:31.6786616Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:10:31.6840711Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:10:31.6890869Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:10:31.6940550Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:10:31.6988225Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:10:31.7038379Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:10:31.7092470Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:10:31.7141524Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:10:31.7196866Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:10:31.7250489Z inflating: build/bin/c10_complex_test 2024-06-26T07:10:31.7301768Z inflating: build/bin/c10_exception_test 2024-06-26T07:10:31.7351018Z inflating: build/bin/c10_flags_test 2024-06-26T07:10:31.7399782Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:10:31.7559510Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:10:31.7608996Z inflating: build/bin/c10_irange_test 2024-06-26T07:10:31.7660720Z inflating: build/bin/c10_lazy_test 2024-06-26T07:10:31.7716387Z inflating: build/bin/c10_logging_test 2024-06-26T07:10:31.7789224Z inflating: build/bin/c10_optional_test 2024-06-26T07:10:31.7850072Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:10:31.7902439Z inflating: build/bin/c10_registry_test 2024-06-26T07:10:31.8048555Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:10:31.8099564Z inflating: build/bin/c10_ssize_test 2024-06-26T07:10:31.8150642Z inflating: build/bin/c10_string_util_test 2024-06-26T07:10:31.8207737Z inflating: build/bin/c10_string_view_test 2024-06-26T07:10:31.8257217Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:10:31.8305322Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:10:31.8361336Z inflating: build/bin/c10_typeid_test 2024-06-26T07:10:31.8796378Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:10:31.9229381Z inflating: build/bin/protoc 2024-06-26T07:10:31.9281526Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:10:31.9332417Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:10:31.9383763Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:10:31.9434285Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:10:31.9485540Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:10:31.9537514Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:10:31.9584565Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:10:31.9635886Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:10:31.9955426Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:10:32.0288499Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:10:32.0634614Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:10:32.0686182Z inflating: build/bin/FileStoreTest 2024-06-26T07:10:32.0740214Z inflating: build/bin/TCPStoreTest 2024-06-26T07:10:32.0792242Z inflating: build/bin/HashStoreTest 2024-06-26T07:10:32.0805216Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:10:32.0858007Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:10:32.0861615Z inflating: build/bin/torch_shm_manager 2024-06-26T07:10:32.0864608Z inflating: build/bin/example_allreduce 2024-06-26T07:10:32.0917975Z inflating: build/bin/test_dist_autograd 2024-06-26T07:10:32.0985452Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:10:32.0988401Z inflating: build/bin/parallel_benchmark 2024-06-26T07:10:32.1053676Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:10:32.1062368Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:10:32.1396457Z inflating: build/bin/test_lazy 2024-06-26T07:10:32.2541870Z inflating: build/bin/test_api 2024-06-26T07:10:32.2613914Z inflating: build/bin/Dict_test 2024-06-26T07:10:32.2663893Z inflating: build/bin/Dimname_test 2024-06-26T07:10:32.2726525Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:10:32.2782166Z inflating: build/bin/NamedTensor_test 2024-06-26T07:10:32.2839596Z inflating: build/bin/apply_utils_test 2024-06-26T07:10:32.2896894Z inflating: build/bin/atest 2024-06-26T07:10:32.2958039Z inflating: build/bin/basic 2024-06-26T07:10:32.3011780Z inflating: build/bin/broadcast_test 2024-06-26T07:10:32.3060429Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:10:32.3116911Z inflating: build/bin/cpu_generator_test 2024-06-26T07:10:32.3168313Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:10:32.3258798Z inflating: build/bin/cpu_rng_test 2024-06-26T07:10:32.3307370Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:10:32.3356173Z inflating: build/bin/dlconvertor_test 2024-06-26T07:10:32.3412710Z inflating: build/bin/extension_backend_test 2024-06-26T07:10:32.3466298Z inflating: build/bin/half_test 2024-06-26T07:10:32.3559599Z inflating: build/bin/ivalue_test 2024-06-26T07:10:32.3607770Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:10:32.3660713Z inflating: build/bin/math_kernel_test 2024-06-26T07:10:32.3713466Z inflating: build/bin/memory_format_test 2024-06-26T07:10:32.3764951Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:10:32.3816858Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:10:32.3870703Z inflating: build/bin/native_test 2024-06-26T07:10:32.3919648Z inflating: build/bin/operator_name_test 2024-06-26T07:10:32.3970066Z inflating: build/bin/operators_test 2024-06-26T07:10:32.4019924Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:10:32.4085111Z inflating: build/bin/pow_test 2024-06-26T07:10:32.4140518Z inflating: build/bin/quantized_test 2024-06-26T07:10:32.4188947Z inflating: build/bin/reduce_ops_test 2024-06-26T07:10:32.4238432Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:10:32.4293282Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:10:32.4349984Z inflating: build/bin/scalar_test 2024-06-26T07:10:32.4400218Z inflating: build/bin/StorageUtils_test 2024-06-26T07:10:32.4451033Z inflating: build/bin/stride_properties_test 2024-06-26T07:10:32.4527699Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:10:32.4579642Z inflating: build/bin/test_parallel 2024-06-26T07:10:32.4582369Z inflating: build/bin/thread_init_test 2024-06-26T07:10:32.4636493Z inflating: build/bin/type_ptr_test 2024-06-26T07:10:32.4694549Z inflating: build/bin/type_test 2024-06-26T07:10:32.4745130Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:10:32.4746594Z inflating: build/bin/verify_api_visibility 2024-06-26T07:10:32.4813979Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:10:32.4864277Z inflating: build/bin/weakref_test 2024-06-26T07:10:32.4914315Z inflating: build/bin/wrapdim_test 2024-06-26T07:10:32.4964936Z inflating: build/bin/xla_tensor_test 2024-06-26T07:10:32.5022639Z inflating: build/bin/IListRef_test 2024-06-26T07:10:32.5125887Z inflating: build/bin/List_test 2024-06-26T07:10:32.5190199Z inflating: build/bin/KernelFunction_test 2024-06-26T07:10:32.5308049Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:10:32.5401797Z inflating: build/bin/kernel_function_test 2024-06-26T07:10:32.5526000Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:10:32.5626538Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:10:32.5685448Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:10:32.5778571Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:10:32.5828158Z inflating: build/bin/CppSignature_test 2024-06-26T07:10:32.5882890Z inflating: build/bin/backend_fallback_test 2024-06-26T07:10:32.5930223Z inflating: build/bin/op_allowlist_test 2024-06-26T07:10:32.5991584Z inflating: build/bin/inline_container_test 2024-06-26T07:10:32.6284014Z inflating: build/bin/op_registration_test 2024-06-26T07:10:32.6336271Z inflating: build/bin/cuda_apply_test 2024-06-26T07:10:32.6385189Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:10:32.6442552Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:10:32.6495518Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:10:32.6563762Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:10:32.6620550Z inflating: build/bin/cuda_complex_test 2024-06-26T07:10:32.6668864Z inflating: build/bin/cuda_device_test 2024-06-26T07:10:32.6724582Z inflating: build/bin/cuda_cub_test 2024-06-26T07:10:32.6774091Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:10:32.6836802Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:10:32.6891745Z inflating: build/bin/cuda_generator_test 2024-06-26T07:10:32.6939880Z inflating: build/bin/cuda_half_test 2024-06-26T07:10:32.6989511Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:10:32.7037441Z inflating: build/bin/cuda_optional_test 2024-06-26T07:10:32.7088001Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:10:32.7138497Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:10:32.7187047Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:10:32.7245866Z inflating: build/bin/cuda_stream_test 2024-06-26T07:10:32.7293849Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:10:32.7344337Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:10:32.7358930Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:10:32.7422513Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:10:32.7478412Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:10:32.7540193Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:10:32.7599930Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:10:32.8406045Z inflating: build/bin/test_tensorexpr 2024-06-26T07:10:32.8965192Z inflating: build/bin/test_jit 2024-06-26T07:10:32.8965627Z creating: .additional_ci_files/ 2024-06-26T07:10:32.9019044Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:10:32.9230894Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:10:32.9256033Z ##[group]Run rm artifacts.zip 2024-06-26T07:10:32.9256424Z rm artifacts.zip 2024-06-26T07:10:32.9266062Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:32.9266567Z env: 2024-06-26T07:10:32.9266862Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:32.9267312Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:32.9267802Z ##[endgroup] 2024-06-26T07:10:32.9980603Z ##[group]Run df -H 2024-06-26T07:10:32.9980927Z df -H 2024-06-26T07:10:32.9988948Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:32.9989454Z env: 2024-06-26T07:10:32.9989737Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:32.9990187Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:32.9990678Z ##[endgroup] 2024-06-26T07:10:33.0016449Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:10:33.0017016Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:10:33.0017607Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:10:33.0018466Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:10:33.0018993Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:10:33.0019480Z /dev/nvme0n1p1 162G 70G 92G 43% / 2024-06-26T07:10:33.0057179Z Prepare all required actions 2024-06-26T07:10:33.0057622Z Getting action download info 2024-06-26T07:10:33.1266936Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:10:33.1267403Z with: 2024-06-26T07:10:33.1267663Z env: 2024-06-26T07:10:33.1267948Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.1268410Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.1268891Z ##[endgroup] 2024-06-26T07:10:33.1299335Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:33.1299782Z with: 2024-06-26T07:10:33.1300051Z name: td_results 2024-06-26T07:10:33.1300376Z s3-bucket: gha-artifacts 2024-06-26T07:10:33.1300737Z region: us-east-1 2024-06-26T07:10:33.1301037Z env: 2024-06-26T07:10:33.1301312Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.1301780Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.1302268Z ##[endgroup] 2024-06-26T07:10:33.5482554Z (node:21943) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:33.5483241Z 2024-06-26T07:10:33.5483508Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:33.5484226Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:33.5485055Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:33.6144961Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:10:33.6151953Z Artifact download has finished successfully 2024-06-26T07:10:33.6298229Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:10:33.6298723Z mkdir -p .additional_ci_files 2024-06-26T07:10:33.6299287Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:10:33.6307395Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:33.6307900Z env: 2024-06-26T07:10:33.6308188Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.6308649Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.6309142Z ##[endgroup] 2024-06-26T07:10:33.6344017Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:10:33.6354604Z ##[error]Process completed with exit code 1. 2024-06-26T07:10:33.6386822Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:10:33.6387299Z .github/scripts/parse_ref.py 2024-06-26T07:10:33.6395047Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:33.6395398Z env: 2024-06-26T07:10:33.6395682Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.6396147Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.6396630Z ##[endgroup] 2024-06-26T07:10:33.6624400Z Prepare all required actions 2024-06-26T07:10:33.6661360Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:10:33.6661968Z with: 2024-06-26T07:10:33.6662515Z github-token: *** 2024-06-26T07:10:33.6662827Z env: 2024-06-26T07:10:33.6663106Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.6663569Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.6664067Z ##[endgroup] 2024-06-26T07:10:33.6679330Z ##[group]Run set -eux 2024-06-26T07:10:33.6679675Z set -eux 2024-06-26T07:10:33.6680279Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:10:33.6687867Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:33.6688361Z env: 2024-06-26T07:10:33.6688643Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:33.6689101Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:33.6689748Z GITHUB_TOKEN: *** 2024-06-26T07:10:33.6690058Z ##[endgroup] 2024-06-26T07:10:33.6709478Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0c2b88ee687943d7d 2024-06-26T07:10:34.9116104Z setting job-id=26688911142 2024-06-26T07:10:34.9116934Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:34.9312374Z Prepare all required actions 2024-06-26T07:10:34.9312826Z Getting action download info 2024-06-26T07:10:35.0390829Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:10:35.0391289Z with: 2024-06-26T07:10:35.0391749Z github-token: *** 2024-06-26T07:10:35.0399886Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:10:35.0408684Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:35.0409370Z env: 2024-06-26T07:10:35.0409650Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:35.0410108Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:35.0410592Z ##[endgroup] 2024-06-26T07:10:35.0450607Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:10:35.0451249Z with: 2024-06-26T07:10:35.0451519Z shell: bash 2024-06-26T07:10:35.0451817Z timeout_minutes: 10 2024-06-26T07:10:35.0452289Z max_attempts: 5 2024-06-26T07:10:35.0452598Z retry_wait_seconds: 30 2024-06-26T07:10:35.0453717Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:35.0454890Z polling_interval_seconds: 1 2024-06-26T07:10:35.0455271Z warning_on_retry: true 2024-06-26T07:10:35.0455611Z continue_on_error: false 2024-06-26T07:10:35.0455950Z env: 2024-06-26T07:10:35.0456233Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:35.0456692Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:35.0457339Z GITHUB_TOKEN: *** 2024-06-26T07:10:35.0457666Z ##[endgroup] 2024-06-26T07:10:35.0958822Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:35.3078714Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:35.3229989Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:10:35.3373678Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:10:35.3381525Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:10:35.3393737Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:10:35.3401270Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:10:35.3600165Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:10:36.0962049Z Command completed after 1 attempt(s). 2024-06-26T07:10:36.1004115Z ##[group]Run set -x 2024-06-26T07:10:36.1004475Z set -x 2024-06-26T07:10:36.1004778Z  2024-06-26T07:10:36.1005347Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:36.1006027Z # in runner workspace 2024-06-26T07:10:36.1006561Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:10:36.1014515Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:36.1015015Z env: 2024-06-26T07:10:36.1015283Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.1015737Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.1016218Z ##[endgroup] 2024-06-26T07:10:36.1036188Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:10:36.1248767Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:36.1249321Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:36.1249804Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:10:36.1250200Z  2024-06-26T07:10:36.1250754Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:36.1251511Z # in runner workspace 2024-06-26T07:10:36.1252142Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:10:36.1252782Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:10:36.1253243Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:10:36.1261865Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:10:36.1270571Z  --selected-test-configs "" \ 2024-06-26T07:10:36.1271037Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:10:36.1271456Z  --tag "${TAG}" \ 2024-06-26T07:10:36.1271860Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:10:36.1272344Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:10:36.1272780Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:10:36.1280362Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:36.1280870Z env: 2024-06-26T07:10:36.1281143Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.1281610Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.1282290Z GITHUB_TOKEN: *** 2024-06-26T07:10:36.1282913Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:36.1283601Z PR_NUMBER: 2024-06-26T07:10:36.1283918Z TAG: ciflow/inductor/129470 2024-06-26T07:10:36.1284294Z EVENT_NAME: push 2024-06-26T07:10:36.1284606Z SCHEDULE: 2024-06-26T07:10:36.1284891Z HEAD_BRANCH: 2024-06-26T07:10:36.1285188Z ##[endgroup] 2024-06-26T07:10:36.1303443Z Workflow: inductor 2024-06-26T07:10:36.1304272Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:36.3822407Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:10:36.5089776Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:10:36.5090261Z echo "Filtered matrix:" 2024-06-26T07:10:36.5098632Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:10:36.5107480Z  2024-06-26T07:10:36.5107861Z echo 2024-06-26T07:10:36.5108241Z echo "Is the current job unstable? False" 2024-06-26T07:10:36.5108707Z  2024-06-26T07:10:36.5108984Z echo 2024-06-26T07:10:36.5109352Z echo "Is keep-going label set? False" 2024-06-26T07:10:36.5109796Z  2024-06-26T07:10:36.5110073Z echo 2024-06-26T07:10:36.5110394Z echo "Renabled issues? " 2024-06-26T07:10:36.5118120Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:36.5118615Z env: 2024-06-26T07:10:36.5118898Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.5119357Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.5119842Z ##[endgroup] 2024-06-26T07:10:36.5139018Z Filtered matrix: 2024-06-26T07:10:36.5149139Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:10:36.5157311Z 2024-06-26T07:10:36.5157460Z Is the current job unstable? False 2024-06-26T07:10:36.5157747Z 2024-06-26T07:10:36.5158031Z Is keep-going label set? False 2024-06-26T07:10:36.5158296Z 2024-06-26T07:10:36.5158424Z Renabled issues? 2024-06-26T07:10:36.5196571Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:36.5197365Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:36.5204231Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:36.5204721Z env: 2024-06-26T07:10:36.5205000Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.5205456Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.5205944Z JOB_TIMEOUT: 240 2024-06-26T07:10:36.5206246Z ##[endgroup] 2024-06-26T07:10:36.5278612Z ##[group]Run set -x 2024-06-26T07:10:36.5279004Z set -x 2024-06-26T07:10:36.5279306Z  2024-06-26T07:10:36.5279661Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:10:36.5280206Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:10:36.5280772Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:10:36.5281286Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:10:36.5281704Z else 2024-06-26T07:10:36.5282057Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:36.5282479Z fi 2024-06-26T07:10:36.5282755Z  2024-06-26T07:10:36.5283222Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:10:36.5283972Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:10:36.5284650Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:10:36.5285250Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:10:36.5285711Z container_name=$(docker run \ 2024-06-26T07:10:36.5286131Z  ${GPU_FLAG:-} \ 2024-06-26T07:10:36.5286501Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:10:36.5286900Z  -e PR_NUMBER \ 2024-06-26T07:10:36.5287258Z  -e GITHUB_ACTIONS \ 2024-06-26T07:10:36.5287651Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:10:36.5288056Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:10:36.5288441Z  -e GITHUB_JOB \ 2024-06-26T07:10:36.5288811Z  -e GITHUB_RUN_ID \ 2024-06-26T07:10:36.5289196Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:10:36.5289604Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:10:36.5289993Z  -e JOB_ID \ 2024-06-26T07:10:36.5290334Z  -e JOB_NAME \ 2024-06-26T07:10:36.5290684Z  -e BASE_SHA \ 2024-06-26T07:10:36.5291105Z  -e BRANCH \ 2024-06-26T07:10:36.5291436Z  -e SHA1 \ 2024-06-26T07:10:36.5291790Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:10:36.5292198Z  -e IN_WHEEL_TEST \ 2024-06-26T07:10:36.5292582Z  -e SHARD_NUMBER \ 2024-06-26T07:10:36.5292948Z  -e TEST_CONFIG \ 2024-06-26T07:10:36.5293320Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:10:36.5293713Z  -e REENABLED_ISSUES \ 2024-06-26T07:10:36.5294125Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:10:36.5294543Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:10:36.5294942Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:10:36.5295328Z  -e NO_TD \ 2024-06-26T07:10:36.5295662Z  -e TD_DISTRIBUTED \ 2024-06-26T07:10:36.5296042Z  -e PR_LABELS \ 2024-06-26T07:10:36.5296451Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:10:36.5296904Z  -e SCCACHE_BUCKET \ 2024-06-26T07:10:36.5297292Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:10:36.5297704Z  -e XLA_CUDA \ 2024-06-26T07:10:36.5298112Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:10:36.5298624Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:10:36.5299142Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:10:36.5299657Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:10:36.5300125Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:10:36.5300543Z  -e DASHBOARD_TAG \ 2024-06-26T07:10:36.5301014Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:10:36.5301571Z  --security-opt seccomp=unconfined \ 2024-06-26T07:10:36.5302181Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:10:36.5302578Z  --ipc=host \ 2024-06-26T07:10:36.5302935Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:10:36.5303335Z  --tty \ 2024-06-26T07:10:36.5303649Z  --detach \ 2024-06-26T07:10:36.5304009Z  --name="${container_name}" \ 2024-06-26T07:10:36.5304572Z  --user jenkins \ 2024-06-26T07:10:36.5305219Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:10:36.5305793Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:10:36.5306231Z  "${DOCKER_IMAGE}" 2024-06-26T07:10:36.5306579Z ) 2024-06-26T07:10:36.5306986Z # Propagate download.pytorch.org IP to container 2024-06-26T07:10:36.5307916Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:10:36.5308888Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:10:36.5309803Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:10:36.5317778Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:36.5318125Z env: 2024-06-26T07:10:36.5318407Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.5318880Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.5319542Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:36.5320043Z PR_NUMBER: 2024-06-26T07:10:36.5320372Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:10:36.5320781Z GITHUB_WORKFLOW: inductor 2024-06-26T07:10:36.5321135Z GITHUB_JOB: test 2024-06-26T07:10:36.5321448Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:10:36.5321801Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:10:36.5322166Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:10:36.5322524Z JOB_ID: 26688911142 2024-06-26T07:10:36.5323148Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:36.5323836Z BRANCH: 2024-06-26T07:10:36.5324168Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:36.5324681Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:36.5325155Z TEST_CONFIG: inductor_timm 2024-06-26T07:10:36.5325511Z SHARD_NUMBER: 1 2024-06-26T07:10:36.5325808Z NUM_TEST_SHARDS: 2 2024-06-26T07:10:36.5326127Z REENABLED_ISSUES: 2024-06-26T07:10:36.5326464Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:10:36.5326844Z VERBOSE_TEST_LOGS: False 2024-06-26T07:10:36.5327201Z NO_TEST_TIMEOUT: False 2024-06-26T07:10:36.5327533Z NO_TD: False 2024-06-26T07:10:36.5327833Z TD_DISTRIBUTED: False 2024-06-26T07:10:36.5328239Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:10:36.5328729Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:10:36.5329108Z SHM_SIZE: 2g 2024-06-26T07:10:36.5330192Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:36.5331423Z XLA_CUDA: 2024-06-26T07:10:36.5331903Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:36.5332511Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:10:36.5332947Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:10:36.5333346Z DASHBOARD_TAG: 2024-06-26T07:10:36.5333851Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:10:36.5334220Z ##[endgroup] 2024-06-26T07:10:36.5353505Z + [[ inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:10:36.5354208Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:10:36.5354727Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:36.5360347Z +++ nproc --ignore=2 2024-06-26T07:10:36.5373042Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:47.8039230Z + container_name=8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T07:10:47.8040367Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:10:47.8041659Z + docker exec -i 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:10:47.8625432Z + echo DOCKER_CONTAINER_ID=8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T07:10:47.8627528Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:47.8629500Z + docker exec -t 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:10:48.2372775Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:10:48.5506681Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:10:48.5511759Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:10:48.5514786Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:10:48.5517855Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:10:48.5521318Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:10:48.5523968Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:10:48.5549973Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:10:48.5609395Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:10:48.6039150Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:10:48.6196328Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:10:49.9493150Z Installing collected packages: torch 2024-06-26T07:10:58.6604625Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:10:58.6606220Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:10:58.6606910Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:10:58.7580376Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:10:58.7584727Z + source .ci/pytorch/common.sh 2024-06-26T07:10:58.7585876Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:10:58.7590742Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:10:58.7592295Z +++ declare -f -t trap_add 2024-06-26T07:10:58.7597633Z ++ set -ex 2024-06-26T07:10:58.7598577Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:10:58.7599133Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:10:58.7599646Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:10:58.7601190Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:10:58.7610641Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:10:58.7611192Z + trap_add cleanup_workspace EXIT 2024-06-26T07:10:58.7611687Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:10:58.7612074Z + shift 2024-06-26T07:10:58.7612357Z + for trap_add_name in "$@" 2024-06-26T07:10:58.7616644Z +++ trap -p EXIT 2024-06-26T07:10:58.7618590Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:10:58.7619080Z +++ extract_trap_cmd 2024-06-26T07:10:58.7619503Z +++ printf '%s\n' '' 2024-06-26T07:10:58.7619969Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:10:58.7620782Z + trap -- ' 2024-06-26T07:10:58.7621180Z cleanup_workspace' EXIT 2024-06-26T07:10:58.7621649Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:10:59.2860873Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:10:59.2873689Z + echo 'Environment variables:' 2024-06-26T07:10:59.2874204Z Environment variables: 2024-06-26T07:10:59.2874575Z + env 2024-06-26T07:10:59.2880091Z INSTALLED_DB=yes 2024-06-26T07:10:59.2880767Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:10:59.2881281Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:10:59.2881826Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:10:59.2882626Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:10:59.2883211Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:10:59.2883703Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:10:59.2884217Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:59.2884796Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:59.2885318Z HOSTNAME=8bd3d1782b65 2024-06-26T07:10:59.2886475Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.2887280Z GITHUB_ACTION=__self 2024-06-26T07:10:59.2887849Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:10:59.2892576Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:10:59.2896476Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:10:59.2896984Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:10:59.2897337Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:10:59.2897682Z TEST_CONFIG=inductor_timm 2024-06-26T07:10:59.2898043Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:10:59.2898512Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:10:59.2898968Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:10:59.2899399Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:10:59.2900018Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:10:59.2900461Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:59.2900956Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:10:59.2901414Z GITHUB_REF_TYPE=tag 2024-06-26T07:10:59.2901740Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:10:59.2902116Z NCCL_VERSION=2.17.1-1 2024-06-26T07:10:59.2902543Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.2902983Z XLA_CUDA= 2024-06-26T07:10:59.2903580Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:10:59.2905706Z *** 2024-06-26T07:10:59.2906000Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:10:59.2906402Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:10:59.2906758Z GITHUB_ACTIONS=true 2024-06-26T07:10:59.2907090Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:59.2907595Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:10:59.2908127Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:10:59.2908585Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.2909092Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:10:59.2909573Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.2910367Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:10:59.2911071Z UCC_HOME=/usr 2024-06-26T07:10:59.2911415Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:10:59.2911815Z VERBOSE_TEST_LOGS=False 2024-06-26T07:10:59.2912156Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:10:59.2912607Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:10:59.2913106Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:10:59.2913568Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:10:59.2913941Z SHARD_NUMBER=1 2024-06-26T07:10:59.2914252Z GITHUB_REF_PROTECTED=false 2024-06-26T07:10:59.2914602Z HOME=/var/lib/jenkins 2024-06-26T07:10:59.2914965Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:10:59.2915397Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:10:59.2915876Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:10:59.2916360Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:10:59.2916729Z CUDA_VERSION=12.1.1 2024-06-26T07:10:59.2917137Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:10:59.2917575Z NUM_TEST_SHARDS=2 2024-06-26T07:10:59.2917874Z UCX_HOME=/usr 2024-06-26T07:10:59.2918374Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:10:59.2919447Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.2920621Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:59.2921795Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.2922896Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:10:59.2923544Z GITHUB_EVENT_NAME=push 2024-06-26T07:10:59.2923877Z DASHBOARD_TAG= 2024-06-26T07:10:59.2924178Z GITHUB_RUN_ID=9673646808 2024-06-26T07:10:59.2924621Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:10:59.2925150Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:10:59.2926133Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.2927022Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:59.2927425Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:10:59.2927849Z PR_NUMBER= 2024-06-26T07:10:59.2928135Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:10:59.2928471Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:10:59.2928904Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:10:59.2929361Z TERM=xterm 2024-06-26T07:10:59.2929691Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:10:59.2930107Z INSTALLED_VISION=yes 2024-06-26T07:10:59.2930410Z BRANCH= 2024-06-26T07:10:59.2930688Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:10:59.2931086Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:10:59.2931744Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:10:59.2932495Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:10:59.2933244Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:10:59.2933729Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:10:59.2934186Z REENABLED_ISSUES= 2024-06-26T07:10:59.2934478Z SHLVL=1 2024-06-26T07:10:59.2934734Z MAX_JOBS=14 2024-06-26T07:10:59.2935132Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:10:59.2935482Z NVARCH=x86_64 2024-06-26T07:10:59.2935776Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:10:59.2936225Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.2936745Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:10:59.2937236Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:10:59.2937931Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:59.2938498Z GITHUB_JOB=test 2024-06-26T07:10:59.2938888Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:10:59.2939440Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:10:59.2939949Z NO_TEST_TIMEOUT=False 2024-06-26T07:10:59.2940273Z TD_DISTRIBUTED=False 2024-06-26T07:10:59.2940636Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:10:59.2941077Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:10:59.2941490Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:10:59.2941855Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:10:59.2942194Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:10:59.2942552Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:10:59.2943590Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:59.2944955Z GITHUB_BASE_REF= 2024-06-26T07:10:59.2945287Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:10:59.2945673Z CI=true 2024-06-26T07:10:59.2945995Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:59.2946408Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:10:59.2946776Z JOB_ID=26688911142 2024-06-26T07:10:59.2947089Z INSTALLED_PROTOBUF=yes 2024-06-26T07:10:59.2947416Z GITHUB_HEAD_REF= 2024-06-26T07:10:59.2947710Z GITHUB_ACTION_REF= 2024-06-26T07:10:59.2948138Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:10:59.2948605Z GITHUB_WORKFLOW=inductor 2024-06-26T07:10:59.2948953Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:10:59.2949854Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.2950660Z NO_TD=False 2024-06-26T07:10:59.2950961Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:10:59.2951317Z _=/usr/bin/env 2024-06-26T07:10:59.2951783Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:10:59.3031594Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:10:59.3032604Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:10:59.3033679Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:10:59.3034556Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:10:59.3035170Z + BUILD_DIR=build 2024-06-26T07:10:59.3035500Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:10:59.3035884Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:10:59.3036230Z + SHARD_NUMBER=1 2024-06-26T07:10:59.3036548Z + NUM_TEST_SHARDS=2 2024-06-26T07:10:59.3036865Z + export VALGRIND=ON 2024-06-26T07:10:59.3037179Z + VALGRIND=ON 2024-06-26T07:10:59.3037665Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:10:59.3038188Z + [[ 0 == \1 ]] 2024-06-26T07:10:59.3038483Z + [[ False == \1 ]] 2024-06-26T07:10:59.3038947Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:10:59.3039477Z ++ realpath build/custom_test_artifacts 2024-06-26T07:10:59.3043450Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:10:59.3044355Z + [[ -n '' ]] 2024-06-26T07:10:59.3044702Z + echo 'Environment variables' 2024-06-26T07:10:59.3045082Z Environment variables 2024-06-26T07:10:59.3045399Z + env 2024-06-26T07:10:59.3048076Z INSTALLED_DB=yes 2024-06-26T07:10:59.3048493Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:10:59.3049019Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:10:59.3049513Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:10:59.3050466Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:10:59.3051235Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:10:59.3051800Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:10:59.3052320Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:59.3052892Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:59.3053382Z HOSTNAME=8bd3d1782b65 2024-06-26T07:10:59.3054217Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.3055020Z GITHUB_ACTION=__self 2024-06-26T07:10:59.3055367Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:10:59.3059328Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:10:59.3064972Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:10:59.3065705Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:10:59.3066215Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:10:59.3066681Z TEST_CONFIG=inductor_timm 2024-06-26T07:10:59.3067178Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:10:59.3067872Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:10:59.3068515Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:10:59.3069136Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:10:59.3069753Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:10:59.3070403Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:59.3071110Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:10:59.3071815Z GITHUB_REF_TYPE=tag 2024-06-26T07:10:59.3072279Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:10:59.3072851Z NCCL_VERSION=2.17.1-1 2024-06-26T07:10:59.3073405Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.3074059Z XLA_CUDA= 2024-06-26T07:10:59.3074698Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:10:59.3075270Z *** 2024-06-26T07:10:59.3075680Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:10:59.3076262Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:10:59.3076791Z GITHUB_ACTIONS=true 2024-06-26T07:10:59.3077275Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:59.3078056Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:10:59.3078812Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:10:59.3079465Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.3080214Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:10:59.3080938Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.3082089Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:10:59.3083151Z UCC_HOME=/usr 2024-06-26T07:10:59.3083637Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:10:59.3084224Z VERBOSE_TEST_LOGS=False 2024-06-26T07:10:59.3084730Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:10:59.3085428Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:10:59.3086177Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:10:59.3087068Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:10:59.3087603Z SHARD_NUMBER=1 2024-06-26T07:10:59.3088056Z GITHUB_REF_PROTECTED=false 2024-06-26T07:10:59.3088596Z HOME=/var/lib/jenkins 2024-06-26T07:10:59.3089129Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:10:59.3114020Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:10:59.3114732Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:10:59.3115589Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:10:59.3116169Z CUDA_VERSION=12.1.1 2024-06-26T07:10:59.3116841Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:10:59.3117480Z NUM_TEST_SHARDS=2 2024-06-26T07:10:59.3117893Z UCX_HOME=/usr 2024-06-26T07:10:59.3118653Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:10:59.3120333Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.3122136Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:59.3124043Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.3125771Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:10:59.3126764Z GITHUB_EVENT_NAME=push 2024-06-26T07:10:59.3127237Z DASHBOARD_TAG= 2024-06-26T07:10:59.3127740Z GITHUB_RUN_ID=9673646808 2024-06-26T07:10:59.3128474Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:10:59.3129306Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:10:59.3130786Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.3132292Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:10:59.3132889Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:10:59.3133443Z PR_NUMBER= 2024-06-26T07:10:59.3133849Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:10:59.3134290Z VALGRIND=ON 2024-06-26T07:10:59.3134668Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:10:59.3135310Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:10:59.3136022Z TERM=xterm 2024-06-26T07:10:59.3136546Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:10:59.3137203Z INSTALLED_VISION=yes 2024-06-26T07:10:59.3137678Z BRANCH= 2024-06-26T07:10:59.3138111Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:10:59.3138744Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:10:59.3139369Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:10:59.3140506Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:10:59.3141648Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:10:59.3142382Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:10:59.3143101Z REENABLED_ISSUES= 2024-06-26T07:10:59.3143539Z SHLVL=1 2024-06-26T07:10:59.3143925Z MAX_JOBS=14 2024-06-26T07:10:59.3144398Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:10:59.3145113Z NVARCH=x86_64 2024-06-26T07:10:59.3145544Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:10:59.3146208Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:59.3147013Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:10:59.3147767Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:10:59.3148763Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:59.3149646Z GITHUB_JOB=test 2024-06-26T07:10:59.3150276Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:10:59.3151105Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:10:59.3151920Z NO_TEST_TIMEOUT=False 2024-06-26T07:10:59.3152423Z TD_DISTRIBUTED=False 2024-06-26T07:10:59.3153000Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:10:59.3153653Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:10:59.3154278Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:10:59.3154824Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:10:59.3155368Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:10:59.3156093Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:10:59.3157677Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:10:59.3159180Z GITHUB_BASE_REF= 2024-06-26T07:10:59.3159592Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:10:59.3160120Z CI=true 2024-06-26T07:10:59.3160604Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:10:59.3161354Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:10:59.3161912Z JOB_ID=26688911142 2024-06-26T07:10:59.3162363Z INSTALLED_PROTOBUF=yes 2024-06-26T07:10:59.3162868Z GITHUB_HEAD_REF= 2024-06-26T07:10:59.3163277Z GITHUB_ACTION_REF= 2024-06-26T07:10:59.3163938Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:10:59.3164592Z GITHUB_WORKFLOW=inductor 2024-06-26T07:10:59.3165042Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:10:59.3166405Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_e5917975-70c8-465a-bc59-2c8489f522ef 2024-06-26T07:10:59.3167671Z NO_TD=False 2024-06-26T07:10:59.3168125Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:10:59.3168663Z _=/usr/bin/env 2024-06-26T07:10:59.3169181Z + echo 'Testing pytorch' 2024-06-26T07:10:59.3169709Z Testing pytorch 2024-06-26T07:10:59.3170207Z + export LANG=C.UTF-8 2024-06-26T07:10:59.3170714Z + LANG=C.UTF-8 2024-06-26T07:10:59.3171134Z + PR_NUMBER= 2024-06-26T07:10:59.3171702Z + [[ inductor_timm == \d\e\f\a\u\l\t ]] 2024-06-26T07:10:59.3172433Z + [[ inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:10:59.3173112Z + [[ inductor_timm == \s\l\o\w ]] 2024-06-26T07:10:59.3174017Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:10:59.3175099Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:10:59.3175964Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:10:59.3176699Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:10:59.3177367Z + [[ inductor_timm == *crossref* ]] 2024-06-26T07:10:59.3178217Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:10:59.3179191Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:10:59.3180177Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:10:59.3181041Z + pip_install --user ninja==1.10.2 2024-06-26T07:10:59.3181890Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:10:59.6764735Z Collecting ninja==1.10.2 2024-06-26T07:10:59.6921042Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:10:59.7051336Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:11:01.0026503Z Installing collected packages: ninja 2024-06-26T07:11:01.0094696Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:11:01.0097198Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:11:01.0137165Z Successfully installed ninja-1.10.2 2024-06-26T07:11:01.1106018Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:01.1108184Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:01.1109763Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:01.1110272Z + install_tlparse 2024-06-26T07:11:01.1110648Z + pip_install --user tlparse==0.3.7 2024-06-26T07:11:01.1111197Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:11:01.4650659Z Collecting tlparse==0.3.7 2024-06-26T07:11:01.4812827Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:11:01.4921837Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:11:02.8275836Z Installing collected packages: tlparse 2024-06-26T07:11:02.8637646Z Successfully installed tlparse-0.3.7 2024-06-26T07:11:02.9594078Z ++ python -m site --user-base 2024-06-26T07:11:02.9747794Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:02.9749685Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:11:02.9750409Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:11:02.9751207Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:02.9752324Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:11:02.9753536Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:11:02.9754254Z + cd test 2024-06-26T07:11:02.9754778Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:11:04.4752244Z + [[ inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:11:04.4752882Z + [[ inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:11:04.4753631Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:04.4754257Z + pushd test 2024-06-26T07:11:04.4754657Z ~/workspace/test ~/workspace 2024-06-26T07:11:04.4755575Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:11:05.9469531Z + CUDA_VERSION=12.1 2024-06-26T07:11:05.9470084Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:11:05.9470441Z + ISCUDA124= 2024-06-26T07:11:05.9470725Z + popd 2024-06-26T07:11:05.9470984Z ~/workspace 2024-06-26T07:11:05.9473334Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:11:05.9474150Z + [[ inductor_timm == *dynamo_eager* ]] 2024-06-26T07:11:05.9474833Z + [[ inductor_timm == *aot_eager* ]] 2024-06-26T07:11:05.9475489Z + [[ inductor_timm == *aot_inductor* ]] 2024-06-26T07:11:05.9476151Z + [[ inductor_timm == *inductor* ]] 2024-06-26T07:11:05.9476762Z + [[ inductor_timm != *perf* ]] 2024-06-26T07:11:05.9477428Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:11:05.9478027Z + [[ inductor_timm == *dynamic* ]] 2024-06-26T07:11:05.9478637Z + [[ inductor_timm == *cpu_inductor* ]] 2024-06-26T07:11:05.9479329Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:11:05.9501152Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:05.9501886Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:11:05.9503924Z + cd test 2024-06-26T07:11:05.9504390Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:11:07.3020992Z PyTorch built with: 2024-06-26T07:11:07.3021613Z - GCC 9.4 2024-06-26T07:11:07.3021964Z - C++ Version: 201703 2024-06-26T07:11:07.3022814Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:07.3023914Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:07.3024875Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:07.3025386Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:11:07.3025870Z - NNPACK is enabled 2024-06-26T07:11:07.3026247Z - CPU capability usage: AVX2 2024-06-26T07:11:07.3026646Z - CUDA Runtime 12.1 2024-06-26T07:11:07.3027172Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:11:07.3027771Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:11:07.3028202Z - Magma 2.6.1 2024-06-26T07:11:07.3035494Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:11:07.3042087Z 2024-06-26T07:11:07.5776919Z + cd test 2024-06-26T07:11:07.5777595Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:11:08.7991150Z ATen/Parallel: 2024-06-26T07:11:08.7991649Z at::get_num_threads() : 8 2024-06-26T07:11:08.7992045Z at::get_num_interop_threads() : 16 2024-06-26T07:11:08.7992593Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:08.7992989Z omp_get_max_threads() : 8 2024-06-26T07:11:08.7993969Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:08.7994768Z mkl_get_max_threads() : 8 2024-06-26T07:11:08.7995375Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:08.7996001Z std::thread::hardware_concurrency() : 16 2024-06-26T07:11:08.7996426Z Environment variables: 2024-06-26T07:11:08.7996761Z OMP_NUM_THREADS : [not set] 2024-06-26T07:11:08.7997132Z MKL_NUM_THREADS : [not set] 2024-06-26T07:11:08.7997505Z ATen parallel backend: OpenMP 2024-06-26T07:11:08.7997765Z 2024-06-26T07:11:09.0426639Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:09.0427272Z + [[ inductor_timm == *backward* ]] 2024-06-26T07:11:09.0427691Z + [[ inductor_timm == *xla* ]] 2024-06-26T07:11:09.0428146Z + [[ inductor_timm == *executorch* ]] 2024-06-26T07:11:09.0428632Z + [[ inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:11:09.0429307Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:09.0429912Z + [[ inductor_timm == distributed ]] 2024-06-26T07:11:09.0430438Z + [[ inductor_timm == *inductor_distributed* ]] 2024-06-26T07:11:09.0430971Z + [[ inductor_timm == *inductor-halide* ]] 2024-06-26T07:11:09.0431569Z + [[ inductor_timm == *inductor-micro-benchmark* ]] 2024-06-26T07:11:09.0432121Z + [[ inductor_timm == *huggingface* ]] 2024-06-26T07:11:09.0432554Z + [[ inductor_timm == *timm* ]] 2024-06-26T07:11:09.0432985Z + install_torchvision 2024-06-26T07:11:09.0433313Z + local orig_preload 2024-06-26T07:11:09.0433675Z + local commit 2024-06-26T07:11:09.0434014Z ++ get_pinned_commit vision 2024-06-26T07:11:09.0434422Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:11:09.0442085Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:09.0442559Z + orig_preload= 2024-06-26T07:11:09.0442964Z + '[' -n '' ']' 2024-06-26T07:11:09.0443849Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:09.0445384Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:09.3604271Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:09.3608615Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-pfl_xhsg 2024-06-26T07:11:09.3627034Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-pfl_xhsg 2024-06-26T07:11:10.7441272Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:11:10.7456153Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:11.9612331Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:12.1953743Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:14.5147459Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:11:14.5200404Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:11:14.5203636Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:11:14.5209630Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:11:14.5420952Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:11:14.5427131Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:11:14.5430309Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:11:14.5433633Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:11:14.5437054Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:11:14.5440057Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:11:14.5877665Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:11:14.6038580Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:11:14.6116129Z Building wheels for collected packages: torchvision 2024-06-26T07:12:27.6139518Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:12:27.6170846Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112805 sha256=9ed0518d8a84ea017bbfe680a079df7dfdeb65c2fd3cdb3f4d1caf29aae24433 2024-06-26T07:12:27.6174296Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:12:27.6208226Z Successfully built torchvision 2024-06-26T07:12:28.8395662Z Installing collected packages: torchvision 2024-06-26T07:12:29.2167093Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:12:29.3441518Z + '[' -n '' ']' 2024-06-26T07:12:29.3442260Z + id=0 2024-06-26T07:12:29.3443070Z + test_dynamo_benchmark timm_models 0 2024-06-26T07:12:29.3444195Z ++ pwd 2024-06-26T07:12:29.3446889Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:12:29.3447714Z + local suite=timm_models 2024-06-26T07:12:29.3448178Z + shift 2024-06-26T07:12:29.3448555Z + local shard_id=0 2024-06-26T07:12:29.3448951Z + shift 2024-06-26T07:12:29.3452975Z + [[ inductor_timm == *perf_compare* ]] 2024-06-26T07:12:29.3453452Z + [[ inductor_timm == *perf* ]] 2024-06-26T07:12:29.3454343Z + [[ inductor_timm == *cpu_inductor* ]] 2024-06-26T07:12:29.3454934Z + [[ inductor_timm == *aot_inductor* ]] 2024-06-26T07:12:29.3455883Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16 2024-06-26T07:12:29.3456659Z ++ pwd 2024-06-26T07:12:29.3457293Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:12:29.3458226Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:12:29.3466592Z + local name=inference 2024-06-26T07:12:29.3466960Z + shift 2024-06-26T07:12:29.3467433Z + local suite=timm_models 2024-06-26T07:12:29.3467892Z + shift 2024-06-26T07:12:29.3468247Z + local shard_id=0 2024-06-26T07:12:29.3468658Z + shift 2024-06-26T07:12:29.3469030Z + partition_flags=() 2024-06-26T07:12:29.3469479Z + local partition_flags 2024-06-26T07:12:29.3469975Z + [[ -n 2 ]] 2024-06-26T07:12:29.3470833Z + [[ -n 0 ]] 2024-06-26T07:12:29.3472203Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:12:29.3472949Z + [[ inductor_timm == *perf_compare* ]] 2024-06-26T07:12:29.3473387Z + [[ inductor_timm == *perf* ]] 2024-06-26T07:12:29.3473786Z + [[ inductor_timm == *aot_inductor* ]] 2024-06-26T07:12:29.3475375Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-06-26T07:12:35.1743511Z 2024-06-26T07:12:35.4852863Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:12:35.4853216Z 2024-06-26T07:12:35.5970150Z model.safetensors: 0% 0.00/95.5M [00:00 will be ignored 2024-06-26T07:31:03.5241787Z E0626 07:31:03.523000 140607695778432 torch/_dynamo/utils.py:1470] RMSE (res-fp64): 0.00937, (ref-fp64): 0.00257 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000 2024-06-26T07:31:03.5245473Z E0626 07:31:03.523000 140607695778432 torch/_dynamo/utils.py:1384] Accuracy failed for key name blocks.0.attn.proj.bias.grad 2024-06-26T07:31:03.5269446Z fail_accuracy 2024-06-26T07:31:03.5767063Z TIMING: entire_frame_compile:63.79031 code_gen:23.96593 inductor_compile:43.80005 backend_compile:53.25682 2024-06-26T07:31:03.5769248Z STATS: call_* op count: 1028 | FakeTensor.__torch_dispatch__:16296 | FakeTensorMode.__torch_dispatch__:94641 | ProxyTorchDispatchMode.__torch_dispatch__:21953 2024-06-26T07:31:03.5770430Z Dynamo produced 3 graphs covering 1028 ops with 7 graph breaks (5 unique) 2024-06-26T07:31:10.3477522Z 2024-06-26T07:31:11.1572142Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:11.1572652Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:11.1573092Z cuda train botnet26t_256 2024-06-26T07:31:56.9459525Z pass 2024-06-26T07:31:56.9562863Z TIMING: entire_frame_compile:26.85753 code_gen:21.25201 inductor_compile:29.91897 backend_compile:23.51653 2024-06-26T07:31:56.9564704Z STATS: call_* op count: 341 | FakeTensor.__torch_dispatch__:5764 | FakeTensorMode.__torch_dispatch__:34145 | ProxyTorchDispatchMode.__torch_dispatch__:9029 2024-06-26T07:31:56.9565857Z Dynamo produced 2 graphs covering 341 ops with 6 graph breaks (5 unique) 2024-06-26T07:32:01.9368260Z 2024-06-26T07:32:05.6817525Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:05.6818019Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:32:05.6818452Z cuda train cait_m36_384 2024-06-26T07:32:05.6823775Z Traceback (most recent call last): 2024-06-26T07:32:05.6824662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T07:32:05.6827156Z self.model_iter_fn(model, example_inputs) 2024-06-26T07:32:05.6828216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in forward_and_backward_pass 2024-06-26T07:32:05.6829040Z pred = mod(*cloned_inputs) 2024-06-26T07:32:05.6830103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:05.6831005Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:05.6832241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:05.6833180Z return forward_call(*args, **kwargs) 2024-06-26T07:32:05.6834735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 361, in forward 2024-06-26T07:32:05.6835766Z x = self.forward_features(x) 2024-06-26T07:32:05.6836936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 346, in forward_features 2024-06-26T07:32:05.6837854Z x = self.blocks(x) 2024-06-26T07:32:05.6839289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:05.6840433Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:05.6841388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:05.6842233Z return forward_call(*args, **kwargs) 2024-06-26T07:32:05.6843174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 249, in forward 2024-06-26T07:32:05.6843995Z input = module(input) 2024-06-26T07:32:05.6844895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:05.6845782Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:05.6846701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:05.6847544Z return forward_call(*args, **kwargs) 2024-06-26T07:32:05.6848384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 197, in forward 2024-06-26T07:32:05.6849264Z x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x))) 2024-06-26T07:32:05.6850334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:05.6851311Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:05.6852278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:05.6853142Z return forward_call(*args, **kwargs) 2024-06-26T07:32:05.6853967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 151, in forward 2024-06-26T07:32:05.6854777Z x = (attn @ v).transpose(1, 2).reshape(B, N, C) 2024-06-26T07:32:05.6857995Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 16.44 MiB is free. Process 3999 has 21.95 GiB memory in use. Of the allocated memory 21.21 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:32:05.6860751Z 2024-06-26T07:32:05.6861056Z The above exception was the direct cause of the following exception: 2024-06-26T07:32:05.6861517Z 2024-06-26T07:32:05.6861661Z Traceback (most recent call last): 2024-06-26T07:32:05.6862339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T07:32:05.6863020Z ) = runner.load_model( 2024-06-26T07:32:05.6863674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 303, in load_model 2024-06-26T07:32:05.6864706Z self.validate_model(model, example_inputs) 2024-06-26T07:32:05.6865491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T07:32:05.6866265Z raise RuntimeError("Eager run failed") from e 2024-06-26T07:32:05.6866729Z RuntimeError: Eager run failed 2024-06-26T07:32:05.6866994Z 2024-06-26T07:32:05.6867111Z eager_fail_to_run 2024-06-26T07:32:09.2069151Z 2024-06-26T07:32:09.9164714Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:09.9165326Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:09.9165866Z cuda train coat_lite_mini 2024-06-26T07:32:58.7055807Z pass 2024-06-26T07:32:58.7168164Z TIMING: entire_frame_compile:24.92011 code_gen:20.79229 inductor_compile:30.84837 backend_compile:22.06716 2024-06-26T07:32:58.7170255Z STATS: call_* op count: 480 | FakeTensor.__torch_dispatch__:6822 | FakeTensorMode.__torch_dispatch__:48378 | ProxyTorchDispatchMode.__torch_dispatch__:14338 2024-06-26T07:32:58.7171510Z Dynamo produced 2 graphs covering 480 ops with 6 graph breaks (5 unique) 2024-06-26T07:33:04.0269044Z 2024-06-26T07:33:05.4847761Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:05.4848235Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:33:05.4848668Z cuda train convit_base 2024-06-26T07:33:05.5174313Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine 2024-06-26T07:34:26.9231264Z pass 2024-06-26T07:34:26.9453022Z TIMING: entire_frame_compile:47.67488 code_gen:28.94818 inductor_compile:48.46353 backend_compile:41.26702 2024-06-26T07:34:26.9455206Z STATS: call_* op count: 1216 | FakeTensor.__torch_dispatch__:16997 | FakeTensorMode.__torch_dispatch__:118277 | ProxyTorchDispatchMode.__torch_dispatch__:40121 2024-06-26T07:34:26.9456370Z Dynamo produced 3 graphs covering 1216 ops with 7 graph breaks (5 unique) 2024-06-26T07:34:33.3104962Z 2024-06-26T07:34:34.0530229Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:34.0530838Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:34.0531302Z cuda train convmixer_768_32 2024-06-26T07:35:22.6229446Z pass 2024-06-26T07:35:22.6378693Z TIMING: entire_frame_compile:26.87927 code_gen:14.67694 inductor_compile:28.15205 backend_compile:24.44036 2024-06-26T07:35:22.6380908Z STATS: call_* op count: 235 | FakeTensor.__torch_dispatch__:12638 | FakeTensorMode.__torch_dispatch__:59863 | ProxyTorchDispatchMode.__torch_dispatch__:15494 2024-06-26T07:35:22.6382066Z Dynamo produced 2 graphs covering 235 ops with 5 graph breaks (4 unique) 2024-06-26T07:35:28.2703629Z 2024-06-26T07:35:29.8294847Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:29.8295380Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:35:29.8298436Z cuda train convnext_base 2024-06-26T07:36:24.5949913Z W0626 07:36:24.594000 140162612503168 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:37:54.8772188Z pass 2024-06-26T07:37:54.9265156Z TIMING: entire_frame_compile:109.47105 code_gen:46.36786 inductor_compile:81.00851 backend_compile:90.77014 2024-06-26T07:37:54.9266572Z STATS: call_* op count: 1621 | FakeTensor.__torch_dispatch__:29605 | FakeTensorMode.__torch_dispatch__:156830 | ProxyTorchDispatchMode.__torch_dispatch__:35464 2024-06-26T07:37:54.9267710Z Dynamo produced 3 graphs covering 1621 ops with 7 graph breaks (5 unique) 2024-06-26T07:38:04.2863899Z 2024-06-26T07:38:04.9445129Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:04.9445611Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:04.9446060Z cuda train crossvit_9_240 2024-06-26T07:38:56.1325467Z W0626 07:38:56.131000 139701874647680 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:40:06.1112092Z pass 2024-06-26T07:40:06.1246535Z TIMING: entire_frame_compile:95.10651 code_gen:39.5905 inductor_compile:68.91981 backend_compile:81.6054 2024-06-26T07:40:06.1249844Z STATS: call_* op count: 1403 | FakeTensor.__torch_dispatch__:25436 | FakeTensorMode.__torch_dispatch__:142105 | ProxyTorchDispatchMode.__torch_dispatch__:34361 2024-06-26T07:40:06.1251007Z Dynamo produced 3 graphs covering 1403 ops with 7 graph breaks (5 unique) 2024-06-26T07:40:14.4952381Z 2024-06-26T07:40:15.5126642Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:15.5127119Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:40:15.5127564Z cuda train cspdarknet53 2024-06-26T07:41:13.2280209Z W0626 07:41:13.227000 139977436717696 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:42:10.3708735Z pass 2024-06-26T07:42:10.3856485Z TIMING: entire_frame_compile:87.31169 code_gen:36.31362 inductor_compile:65.36329 backend_compile:72.20721 2024-06-26T07:42:10.3857789Z STATS: call_* op count: 1070 | FakeTensor.__torch_dispatch__:22900 | FakeTensorMode.__torch_dispatch__:115291 | ProxyTorchDispatchMode.__torch_dispatch__:24624 2024-06-26T07:42:10.3859159Z Dynamo produced 3 graphs covering 1070 ops with 7 graph breaks (5 unique) 2024-06-26T07:42:18.7032865Z 2024-06-26T07:42:20.1254267Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:20.1254758Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:42:20.1255214Z cuda train deit_base_distilled_patch16_224 2024-06-26T07:42:47.4880230Z W0626 07:42:47.487000 140040989475456 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:43:26.9315125Z pass 2024-06-26T07:43:26.9832510Z TIMING: entire_frame_compile:49.94262 code_gen:19.86909 inductor_compile:35.89242 backend_compile:42.49862 2024-06-26T07:43:26.9835124Z STATS: call_* op count: 841 | FakeTensor.__torch_dispatch__:14206 | FakeTensorMode.__torch_dispatch__:76972 | ProxyTorchDispatchMode.__torch_dispatch__:18136 2024-06-26T07:43:26.9837363Z Dynamo produced 3 graphs covering 841 ops with 7 graph breaks (5 unique) 2024-06-26T07:43:33.0547002Z 2024-06-26T07:43:34.2013253Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:43:34.2013797Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:43:34.2014351Z cuda train dla102 2024-06-26T07:44:49.1178551Z W0626 07:44:49.116000 139651911680640 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:46:18.4940309Z pass 2024-06-26T07:46:18.5153108Z TIMING: entire_frame_compile:127.62905 code_gen:56.5851 inductor_compile:96.22008 backend_compile:108.54006 2024-06-26T07:46:18.5154426Z STATS: call_* op count: 1389 | FakeTensor.__torch_dispatch__:34853 | FakeTensorMode.__torch_dispatch__:173859 | ProxyTorchDispatchMode.__torch_dispatch__:36994 2024-06-26T07:46:18.5155593Z Dynamo produced 3 graphs covering 1389 ops with 7 graph breaks (5 unique) 2024-06-26T07:46:29.3749128Z 2024-06-26T07:46:30.8674974Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:30.8675430Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:46:30.8675868Z cuda train dm_nfnet_f0 2024-06-26T07:47:25.2566535Z pass 2024-06-26T07:47:25.3068531Z TIMING: entire_frame_compile:28.41836 code_gen:17.15187 inductor_compile:30.90895 backend_compile:21.56498 2024-06-26T07:47:25.3071033Z STATS: call_* op count: 620 | FakeTensorMode.__torch_dispatch__:57475 | FakeTensor.__torch_dispatch__:11878 | ProxyTorchDispatchMode.__torch_dispatch__:16005 2024-06-26T07:47:25.3072197Z Dynamo produced 2 graphs covering 620 ops with 6 graph breaks (5 unique) 2024-06-26T07:47:30.7729640Z 2024-06-26T07:47:32.1138747Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:47:32.1139247Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:47:32.1139690Z cuda train dpn107 2024-06-26T07:49:10.4356276Z pass 2024-06-26T07:49:10.4964167Z TIMING: entire_frame_compile:60.1753 code_gen:33.58517 inductor_compile:55.32212 backend_compile:49.59222 2024-06-26T07:49:10.4967494Z STATS: call_* op count: 746 | FakeTensor.__torch_dispatch__:18755 | FakeTensorMode.__torch_dispatch__:102809 | ProxyTorchDispatchMode.__torch_dispatch__:28636 2024-06-26T07:49:10.4969682Z Dynamo produced 2 graphs covering 746 ops with 6 graph breaks (5 unique) 2024-06-26T07:49:17.7263351Z 2024-06-26T07:49:18.5013281Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:18.5013751Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:18.5016403Z cuda train eca_botnext26ts_256 2024-06-26T07:50:04.2912476Z W0626 07:50:04.290000 139706439541376 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:50:35.3181033Z pass 2024-06-26T07:50:35.3295060Z TIMING: entire_frame_compile:54.17763 code_gen:29.63268 inductor_compile:46.89645 backend_compile:45.85878 2024-06-26T07:50:35.3296429Z STATS: call_* op count: 707 | FakeTensor.__torch_dispatch__:12859 | FakeTensorMode.__torch_dispatch__:67202 | ProxyTorchDispatchMode.__torch_dispatch__:15134 2024-06-26T07:50:35.3297553Z Dynamo produced 3 graphs covering 707 ops with 7 graph breaks (5 unique) 2024-06-26T07:50:41.4024783Z 2024-06-26T07:50:42.2189441Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:50:42.2189921Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:50:42.2190357Z cuda train eca_halonext26ts 2024-06-26T07:51:32.0650696Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T07:51:32.0652074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T07:51:32.0655076Z pred = mod(*cloned_inputs) 2024-06-26T07:51:32.0656370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:32.0657295Z return forward_call(*args, **kwargs) 2024-06-26T07:51:32.0658509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1257, in forward 2024-06-26T07:51:32.0659534Z x = self.forward_features(x) 2024-06-26T07:51:32.0660464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1249, in forward_features 2024-06-26T07:51:32.0661291Z x = self.stages(x) 2024-06-26T07:51:32.0662120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:32.0663011Z return forward_call(*args, **kwargs) 2024-06-26T07:51:32.0663872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 883, in forward 2024-06-26T07:51:32.0664914Z x = self.self_attn(x) 2024-06-26T07:51:32.0665766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:51:32.0666606Z return forward_call(*args, **kwargs) 2024-06-26T07:51:32.0667478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/layers/halo_attn.py", line 191, in forward 2024-06-26T07:51:32.0668564Z kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape( 2024-06-26T07:51:32.0669152Z 2024-06-26T07:51:32.3199805Z W0626 07:51:32.319000 140145619579520 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:52:03.3038094Z pass 2024-06-26T07:52:03.3104810Z TIMING: entire_frame_compile:56.4118 code_gen:32.68396 inductor_compile:50.7985 backend_compile:47.19789 2024-06-26T07:52:03.3106102Z STATS: call_* op count: 740 | FakeTensor.__torch_dispatch__:12680 | FakeTensorMode.__torch_dispatch__:69484 | ProxyTorchDispatchMode.__torch_dispatch__:15740 2024-06-26T07:52:03.3107239Z Dynamo produced 3 graphs covering 740 ops with 7 graph breaks (5 unique) 2024-06-26T07:52:09.4736662Z 2024-06-26T07:52:10.1837270Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:10.1837751Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:10.1838185Z cuda train ese_vovnet19b_dw 2024-06-26T07:52:41.3617017Z W0626 07:52:41.360000 139926551978624 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:53:10.9463571Z pass 2024-06-26T07:53:10.9534128Z TIMING: entire_frame_compile:42.66429 code_gen:24.9856 inductor_compile:37.16836 backend_compile:35.49418 2024-06-26T07:53:10.9537332Z STATS: call_* op count: 459 | FakeTensor.__torch_dispatch__:9729 | FakeTensorMode.__torch_dispatch__:49643 | ProxyTorchDispatchMode.__torch_dispatch__:10392 2024-06-26T07:53:10.9538538Z Dynamo produced 4 graphs covering 459 ops with 7 graph breaks (5 unique) 2024-06-26T07:53:16.6215376Z 2024-06-26T07:53:17.3464238Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:17.3464963Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:17.3465429Z cuda train fbnetc_100 2024-06-26T07:54:05.4116820Z W0626 07:54:05.410000 139680548344448 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:55:02.9731440Z pass 2024-06-26T07:55:02.9777838Z TIMING: entire_frame_compile:83.13302 code_gen:33.61927 inductor_compile:59.33776 backend_compile:69.05098 2024-06-26T07:55:02.9780028Z STATS: call_* op count: 993 | FakeTensor.__torch_dispatch__:20728 | FakeTensorMode.__torch_dispatch__:105778 | ProxyTorchDispatchMode.__torch_dispatch__:22083 2024-06-26T07:55:02.9781194Z Dynamo produced 4 graphs covering 993 ops with 7 graph breaks (5 unique) 2024-06-26T07:55:11.0573151Z 2024-06-26T07:55:11.9423708Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:11.9424175Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:11.9427313Z cuda train fbnetv3_b 2024-06-26T07:56:37.2874917Z pass 2024-06-26T07:56:37.2956480Z TIMING: entire_frame_compile:50.61873 code_gen:31.90209 inductor_compile:55.93889 backend_compile:42.87299 2024-06-26T07:56:37.2957801Z STATS: call_* op count: 610 | FakeTensor.__torch_dispatch__:20660 | FakeTensorMode.__torch_dispatch__:91852 | ProxyTorchDispatchMode.__torch_dispatch__:23345 2024-06-26T07:56:37.2958940Z Dynamo produced 2 graphs covering 610 ops with 6 graph breaks (5 unique) 2024-06-26T07:56:44.9239572Z 2024-06-26T07:56:45.9466309Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:56:45.9467027Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:56:45.9467646Z cuda train gernet_l 2024-06-26T07:57:29.2873430Z pass 2024-06-26T07:57:29.3074994Z TIMING: entire_frame_compile:27.62769 code_gen:13.71482 inductor_compile:24.90087 backend_compile:22.25937 2024-06-26T07:57:29.3077131Z STATS: call_* op count: 410 | FakeTensor.__torch_dispatch__:9258 | FakeTensorMode.__torch_dispatch__:47041 | ProxyTorchDispatchMode.__torch_dispatch__:12116 2024-06-26T07:57:29.3078309Z Dynamo produced 2 graphs covering 410 ops with 6 graph breaks (5 unique) 2024-06-26T07:57:34.7122884Z 2024-06-26T07:57:35.5216705Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:57:35.5217188Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:57:35.5217628Z cuda train ghostnet_100 2024-06-26T07:58:43.1532314Z pass 2024-06-26T07:58:43.1578433Z TIMING: entire_frame_compile:41.62872 code_gen:27.78972 inductor_compile:46.30054 backend_compile:37.83631 2024-06-26T07:58:43.1580808Z STATS: call_* op count: 360 | FakeTensor.__torch_dispatch__:15561 | FakeTensorMode.__torch_dispatch__:77583 | ProxyTorchDispatchMode.__torch_dispatch__:19864 2024-06-26T07:58:43.1582034Z Dynamo produced 2 graphs covering 360 ops with 6 graph breaks (5 unique) 2024-06-26T07:58:49.8624340Z 2024-06-26T07:58:50.9792150Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:58:50.9792674Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:58:50.9795602Z cuda train gluon_inception_v3 2024-06-26T07:59:22.7100367Z W0626 07:59:22.709000 139736264233600 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:00:39.2272139Z pass 2024-06-26T08:00:39.2499880Z TIMING: entire_frame_compile:96.23961 inductor_compile:44.7505 backend_compile:75.25362 code_gen:21.02149 2024-06-26T08:00:39.2502267Z STATS: call_* op count: 1376 | FakeTensor.__torch_dispatch__:17669 | FakeTensorMode.__torch_dispatch__:126519 | ProxyTorchDispatchMode.__torch_dispatch__:33213 2024-06-26T08:00:39.2503447Z Dynamo produced 3 graphs covering 1376 ops with 7 graph breaks (5 unique) 2024-06-26T08:00:46.7589723Z 2024-06-26T08:00:47.5805353Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:00:47.5805823Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:00:47.5809553Z cuda train gmixer_24_224 2024-06-26T08:01:36.6697049Z pass 2024-06-26T08:01:36.6942761Z TIMING: entire_frame_compile:26.69366 code_gen:16.06355 inductor_compile:28.41181 backend_compile:23.58376 2024-06-26T08:01:36.6944086Z STATS: call_* op count: 684 | FakeTensor.__torch_dispatch__:12879 | FakeTensorMode.__torch_dispatch__:71357 | ProxyTorchDispatchMode.__torch_dispatch__:23778 2024-06-26T08:01:36.6945477Z Dynamo produced 2 graphs covering 684 ops with 6 graph breaks (5 unique) 2024-06-26T08:01:42.2367308Z 2024-06-26T08:01:42.9985090Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:42.9985716Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:42.9986346Z cuda train gmlp_s16_224 2024-06-26T08:02:30.9400010Z W0626 08:02:30.939000 139880901431936 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:03:46.5508968Z pass 2024-06-26T08:03:46.5709197Z TIMING: entire_frame_compile:95.31041 code_gen:36.08833 inductor_compile:68.65813 backend_compile:80.79511 2024-06-26T08:03:46.5710537Z STATS: call_* op count: 1458 | FakeTensor.__torch_dispatch__:28954 | FakeTensorMode.__torch_dispatch__:150087 | ProxyTorchDispatchMode.__torch_dispatch__:35364 2024-06-26T08:03:46.5711670Z Dynamo produced 3 graphs covering 1458 ops with 7 graph breaks (5 unique) 2024-06-26T08:03:55.4274728Z 2024-06-26T08:03:57.0421035Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:03:57.0421513Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:03:57.0421982Z cuda train hrnet_w18 2024-06-26T08:07:04.0081305Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:07:04.0082966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:07:04.0083970Z pred = mod(*cloned_inputs) 2024-06-26T08:07:04.0085074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:04.0085925Z return forward_call(*args, **kwargs) 2024-06-26T08:07:04.0086950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 792, in forward 2024-06-26T08:07:04.0088061Z y = self.forward_features(x) 2024-06-26T08:07:04.0089100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 770, in forward_features 2024-06-26T08:07:04.0090003Z yl = self.stages(x) 2024-06-26T08:07:04.0090901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 751, in stages 2024-06-26T08:07:04.0091881Z yl = self.stage2(xl) 2024-06-26T08:07:04.0093037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:04.0094199Z return forward_call(*args, **kwargs) 2024-06-26T08:07:04.0095338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 506, in forward 2024-06-26T08:07:04.0096253Z x = module(x) 2024-06-26T08:07:04.0097092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:07:04.0097937Z return forward_call(*args, **kwargs) 2024-06-26T08:07:04.0098787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 484, in forward 2024-06-26T08:07:04.0099525Z y = y + f(x[j]) 2024-06-26T08:07:04.0099729Z 2024-06-26T08:07:07.9899432Z pass 2024-06-26T08:07:08.0080993Z TIMING: entire_frame_compile:124.41367 code_gen:64.4732 inductor_compile:121.17471 backend_compile:112.71189 2024-06-26T08:07:08.0082371Z STATS: call_* op count: 1394 | FakeTensor.__torch_dispatch__:52796 | FakeTensorMode.__torch_dispatch__:272271 | ProxyTorchDispatchMode.__torch_dispatch__:71274 2024-06-26T08:07:08.0083514Z Dynamo produced 2 graphs covering 1394 ops with 5 graph breaks (4 unique) 2024-06-26T08:07:21.3125478Z 2024-06-26T08:07:22.4179269Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:07:22.4179851Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:07:22.4180743Z cuda train inception_v3 2024-06-26T08:07:55.3704228Z pass 2024-06-26T08:07:55.3911949Z TIMING: entire_frame_compile:24.89759 inductor_compile:5.53268 backend_compile:16.83393 2024-06-26T08:07:55.3913167Z STATS: call_* op count: 506 | FakeTensor.__torch_dispatch__:2985 | FakeTensorMode.__torch_dispatch__:50578 | ProxyTorchDispatchMode.__torch_dispatch__:20409 2024-06-26T08:07:55.3915550Z Dynamo produced 2 graphs covering 506 ops with 6 graph breaks (5 unique) 2024-06-26T08:07:59.7422750Z 2024-06-26T08:08:01.0995130Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:08:01.0995595Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:08:01.0996030Z cuda train jx_nest_base 2024-06-26T08:09:16.5681796Z W0626 08:09:16.567000 140321009738368 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:10:34.6158541Z pass 2024-06-26T08:10:34.6613960Z TIMING: entire_frame_compile:114.23876 code_gen:48.89947 inductor_compile:84.17971 backend_compile:97.00236 2024-06-26T08:10:34.6615299Z STATS: call_* op count: 1700 | FakeTensor.__torch_dispatch__:29051 | FakeTensorMode.__torch_dispatch__:170517 | ProxyTorchDispatchMode.__torch_dispatch__:42081 2024-06-26T08:10:34.6616438Z Dynamo produced 3 graphs covering 1700 ops with 7 graph breaks (5 unique) 2024-06-26T08:10:43.9581787Z 2024-06-26T08:10:44.5393022Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:44.5393522Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:44.5393965Z cuda train lcnet_050 2024-06-26T08:11:12.3554158Z pass 2024-06-26T08:11:12.3577945Z TIMING: entire_frame_compile:16.1639 code_gen:10.19612 inductor_compile:18.09097 backend_compile:13.72703 2024-06-26T08:11:12.3579238Z STATS: call_* op count: 168 | FakeTensor.__torch_dispatch__:6241 | FakeTensorMode.__torch_dispatch__:27828 | ProxyTorchDispatchMode.__torch_dispatch__:6908 2024-06-26T08:11:12.3583105Z Dynamo produced 2 graphs covering 168 ops with 6 graph breaks (5 unique) 2024-06-26T08:11:17.0904270Z 2024-06-26T08:11:17.9616749Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:17.9617461Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:17.9618123Z cuda train levit_128 2024-06-26T08:12:27.4085055Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:12:27.4086253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:12:27.4090167Z pred = mod(*cloned_inputs) 2024-06-26T08:12:27.4091789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:27.4093093Z return forward_call(*args, **kwargs) 2024-06-26T08:12:27.4094463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 652, in forward 2024-06-26T08:12:27.4095660Z x = self.forward_features(x) 2024-06-26T08:12:27.4097037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 643, in forward_features 2024-06-26T08:12:27.4098305Z x = self.stages(x) 2024-06-26T08:12:27.4099571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:27.4100905Z return forward_call(*args, **kwargs) 2024-06-26T08:12:27.4102229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 518, in forward 2024-06-26T08:12:27.4103332Z x = self.blocks(x) 2024-06-26T08:12:27.4105178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:27.4106106Z return forward_call(*args, **kwargs) 2024-06-26T08:12:27.4107003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 456, in forward 2024-06-26T08:12:27.4107788Z x = x + self.drop_path1(self.attn(x)) 2024-06-26T08:12:27.4108992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:12:27.4109832Z return forward_call(*args, **kwargs) 2024-06-26T08:12:27.4110704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 236, in forward 2024-06-26T08:12:27.4111620Z attn = q @ k * self.scale + self.get_attention_biases(x.device) 2024-06-26T08:12:27.4112762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 211, in get_attention_biases 2024-06-26T08:12:27.4113704Z return self.attention_biases[:, self.attention_bias_idxs] 2024-06-26T08:12:27.4114101Z 2024-06-26T08:12:27.5618222Z W0626 08:12:27.561000 140528278381184 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:13:21.7052236Z pass 2024-06-26T08:13:21.7087973Z TIMING: entire_frame_compile:93.85691 code_gen:41.83179 inductor_compile:72.89712 backend_compile:80.96624 2024-06-26T08:13:21.7089691Z STATS: call_* op count: 1225 | FakeTensor.__torch_dispatch__:24276 | FakeTensorMode.__torch_dispatch__:135341 | ProxyTorchDispatchMode.__torch_dispatch__:32073 2024-06-26T08:13:21.7090855Z Dynamo produced 3 graphs covering 1225 ops with 7 graph breaks (5 unique) 2024-06-26T08:13:30.7505205Z 2024-06-26T08:13:31.8383512Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:31.8384253Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:13:31.8385188Z cuda train mixer_b16_224 2024-06-26T08:14:00.1720661Z W0626 08:14:00.171000 139877564756608 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:14:38.6277019Z pass 2024-06-26T08:14:38.6695677Z TIMING: entire_frame_compile:50.35416 code_gen:20.16313 inductor_compile:36.05036 backend_compile:43.37571 2024-06-26T08:14:38.6698593Z STATS: call_* op count: 720 | FakeTensor.__torch_dispatch__:14404 | FakeTensorMode.__torch_dispatch__:74409 | ProxyTorchDispatchMode.__torch_dispatch__:17844 2024-06-26T08:14:38.6699769Z Dynamo produced 3 graphs covering 720 ops with 7 graph breaks (5 unique) 2024-06-26T08:14:41.9253512Z accuracy pass_rate=93.33% 2024-06-26T08:14:41.9257500Z calls_captured gmean=0.00x mean=833.533x 2024-06-26T08:14:41.9261041Z unique_graphs gmean=0.00x mean=2.533x 2024-06-26T08:14:41.9265504Z graph_breaks gmean=0.00x mean=6.267x 2024-06-26T08:14:41.9269564Z unique_graph_breaks gmean=0.00x mean=4.767x 2024-06-26T08:14:41.9273706Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:14:41.9277790Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:14:41.9281681Z cudagraph_skips gmean=0.00x mean=0.133x 2024-06-26T08:14:43.4687025Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_timm_training.csv 2024-06-26T08:14:43.7270383Z adv_inception_v3 PASS 2024-06-26T08:14:43.7275161Z beit_base_patch16_224 XFAIL 2024-06-26T08:14:43.7279292Z botnet26t_256 PASS 2024-06-26T08:14:43.7283781Z cait_m36_384 XFAIL 2024-06-26T08:14:43.7288233Z coat_lite_mini PASS 2024-06-26T08:14:43.7292886Z convit_base PASS 2024-06-26T08:14:43.7297282Z convmixer_768_32 PASS 2024-06-26T08:14:43.7301898Z convnext_base PASS 2024-06-26T08:14:43.7306365Z crossvit_9_240 PASS 2024-06-26T08:14:43.7310776Z cspdarknet53 PASS 2024-06-26T08:14:43.7316430Z deit_base_distilled_patch16_224 PASS 2024-06-26T08:14:43.7319536Z dla102 PASS 2024-06-26T08:14:43.7323938Z dm_nfnet_f0 PASS 2024-06-26T08:14:43.7328221Z dpn107 PASS 2024-06-26T08:14:43.7332750Z eca_botnext26ts_256 PASS 2024-06-26T08:14:43.7337286Z eca_halonext26ts PASS 2024-06-26T08:14:43.7341783Z ese_vovnet19b_dw PASS 2024-06-26T08:14:43.7346243Z fbnetc_100 PASS 2024-06-26T08:14:43.7350707Z fbnetv3_b PASS 2024-06-26T08:14:43.7354982Z gernet_l PASS 2024-06-26T08:14:43.7359466Z ghostnet_100 PASS 2024-06-26T08:14:43.7363842Z gluon_inception_v3 PASS 2024-06-26T08:14:43.7368537Z gmixer_24_224 PASS 2024-06-26T08:14:43.7372688Z gmlp_s16_224 PASS 2024-06-26T08:14:43.7377330Z hrnet_w18 PASS 2024-06-26T08:14:43.7381587Z inception_v3 PASS 2024-06-26T08:14:43.7386157Z jx_nest_base PASS 2024-06-26T08:14:43.7390507Z lcnet_050 PASS 2024-06-26T08:14:43.7394979Z levit_128 PASS 2024-06-26T08:14:43.7399397Z mixer_b16_224 PASS 2024-06-26T08:14:43.7771611Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_timm_training.csv 2024-06-26T08:14:44.0560994Z adv_inception_v3 PASS 2024-06-26T08:14:44.0564449Z beit_base_patch16_224 PASS 2024-06-26T08:14:44.0569346Z botnet26t_256 PASS 2024-06-26T08:14:44.0574393Z cait_m36_384 PASS 2024-06-26T08:14:44.0579347Z coat_lite_mini PASS 2024-06-26T08:14:44.0584139Z convit_base PASS 2024-06-26T08:14:44.0589369Z convmixer_768_32 PASS 2024-06-26T08:14:44.0594144Z convnext_base PASS 2024-06-26T08:14:44.0599009Z crossvit_9_240 PASS 2024-06-26T08:14:44.0603813Z cspdarknet53 PASS 2024-06-26T08:14:44.0608714Z deit_base_distilled_patch16_224 PASS 2024-06-26T08:14:44.0613727Z dla102 PASS 2024-06-26T08:14:44.0618299Z dm_nfnet_f0 PASS 2024-06-26T08:14:44.0623179Z dpn107 PASS 2024-06-26T08:14:44.0628206Z eca_botnext26ts_256 PASS 2024-06-26T08:14:44.0633000Z eca_halonext26ts PASS 2024-06-26T08:14:44.0637787Z ese_vovnet19b_dw PASS 2024-06-26T08:14:44.0642541Z fbnetc_100 PASS 2024-06-26T08:14:44.0647440Z fbnetv3_b PASS 2024-06-26T08:14:44.0652262Z gernet_l PASS 2024-06-26T08:14:44.0657103Z ghostnet_100 PASS 2024-06-26T08:14:44.0662013Z gluon_inception_v3 PASS 2024-06-26T08:14:44.0666965Z gmixer_24_224 PASS 2024-06-26T08:14:44.0671694Z gmlp_s16_224 PASS 2024-06-26T08:14:44.0676529Z hrnet_w18 PASS 2024-06-26T08:14:44.0681326Z inception_v3 PASS 2024-06-26T08:14:44.0686110Z jx_nest_base PASS 2024-06-26T08:14:44.0691003Z lcnet_050 PASS 2024-06-26T08:14:44.0696017Z levit_128 PASS 2024-06-26T08:14:44.0700962Z mixer_b16_224 PASS 2024-06-26T08:14:44.1097222Z + cleanup_workspace 2024-06-26T08:14:44.1098337Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:14:44.1099710Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:14:44.1100864Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:14:44.1101747Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:14:44.1102696Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:14:44.1103581Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:14:44.1104870Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:14:44.5641233Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:14:44.5641791Z cat test/**/*_toprint.log || true 2024-06-26T08:14:44.5649514Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:44.5650019Z env: 2024-06-26T08:14:44.5650292Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.5650745Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.5651582Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.5652222Z ##[endgroup] 2024-06-26T08:14:44.5714176Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:14:44.5742132Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:14:44.5742596Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:14:44.5750550Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:44.5751041Z env: 2024-06-26T08:14:44.5751315Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.5751776Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.5752509Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.5753150Z MONITOR_SCRIPT_PID: 21889 2024-06-26T08:14:44.5753512Z ##[endgroup] 2024-06-26T08:14:44.5937858Z Prepare all required actions 2024-06-26T08:14:44.5938301Z Getting action download info 2024-06-26T08:14:44.7555858Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:14:44.9131785Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:14:44.9132250Z with: 2024-06-26T08:14:44.9132779Z file-suffix: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142 2024-06-26T08:14:44.9133426Z s3-bucket: gha-artifacts 2024-06-26T08:14:44.9133761Z env: 2024-06-26T08:14:44.9134038Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.9134489Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.9135236Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.9135866Z ##[endgroup] 2024-06-26T08:14:44.9160191Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:14:44.9160854Z # Remove any previous test jsons if they exist 2024-06-26T08:14:44.9161373Z rm -f test-jsons-*.zip 2024-06-26T08:14:44.9161896Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:14:44.9169743Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:44.9170243Z env: 2024-06-26T08:14:44.9170523Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.9171032Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.9171770Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.9172673Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142 2024-06-26T08:14:44.9173287Z ##[endgroup] 2024-06-26T08:14:44.9350869Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:14:44.9379978Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:14:44.9380731Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:14:44.9386976Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:14:44.9412693Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:14:44.9413356Z # Remove any previous test reports if they exist 2024-06-26T08:14:44.9413892Z rm -f test-reports-*.zip 2024-06-26T08:14:44.9414494Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:14:44.9422034Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:44.9422527Z env: 2024-06-26T08:14:44.9422809Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.9423272Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.9424132Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.9425393Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142 2024-06-26T08:14:44.9426122Z ##[endgroup] 2024-06-26T08:14:44.9589732Z adding: test/test-reports/inference_timm_models.csv (deflated 69%) 2024-06-26T08:14:44.9590535Z adding: test/test-reports/training_timm_models.csv (deflated 65%) 2024-06-26T08:14:44.9594076Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-06-26T08:14:44.9595018Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%) 2024-06-26T08:14:44.9620092Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:14:44.9620725Z # Remove any previous usage logs if they exist 2024-06-26T08:14:44.9621236Z rm -f logs-*.zip 2024-06-26T08:14:44.9621918Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:14:44.9622723Z # so check to see if the file exists first 2024-06-26T08:14:44.9623224Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:14:44.9623762Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:14:44.9624258Z fi 2024-06-26T08:14:44.9624990Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:14:44.9625579Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:14:44.9626082Z fi 2024-06-26T08:14:44.9633372Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:44.9633880Z env: 2024-06-26T08:14:44.9634155Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.9634621Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.9635366Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.9636225Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142 2024-06-26T08:14:44.9636831Z ##[endgroup] 2024-06-26T08:14:44.9700228Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:14:44.9765771Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:14:44.9766490Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:14:44.9767058Z rm -f debug-*.zip 2024-06-26T08:14:44.9767444Z if [ -d 'test/debug' ]; then 2024-06-26T08:14:44.9767945Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:14:44.9768426Z fi 2024-06-26T08:14:44.9775924Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:44.9776424Z env: 2024-06-26T08:14:44.9776704Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.9777159Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.9777885Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.9778743Z FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142 2024-06-26T08:14:44.9779365Z ##[endgroup] 2024-06-26T08:14:44.9852761Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:14:44.9853204Z with: 2024-06-26T08:14:44.9853496Z s3-bucket: gha-artifacts 2024-06-26T08:14:44.9853934Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:14:44.9854430Z retention-days: 14 2024-06-26T08:14:44.9854773Z if-no-files-found: warn 2024-06-26T08:14:44.9855134Z path: test-jsons-*.zip 2024-06-26T08:14:44.9855478Z name: artifact 2024-06-26T08:14:44.9855777Z region: us-east-1 2024-06-26T08:14:44.9856076Z env: 2024-06-26T08:14:44.9856356Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:44.9856803Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:44.9857525Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:44.9858162Z ##[endgroup] 2024-06-26T08:14:45.2888000Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:14:45.2889195Z With the provided path, there will be 1 file uploaded 2024-06-26T08:14:45.2890801Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:14:45.2915240Z Starting upload of test-jsons-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142.zip 2024-06-26T08:14:45.4941588Z Finished upload of test-jsons-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142.zip 2024-06-26T08:14:45.5085528Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:14:45.5085969Z with: 2024-06-26T08:14:45.5086242Z s3-bucket: gha-artifacts 2024-06-26T08:14:45.5086672Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:14:45.5087147Z retention-days: 14 2024-06-26T08:14:45.5087481Z if-no-files-found: error 2024-06-26T08:14:45.5087836Z path: test-reports-*.zip 2024-06-26T08:14:45.5088186Z name: artifact 2024-06-26T08:14:45.5088485Z region: us-east-1 2024-06-26T08:14:45.5088779Z env: 2024-06-26T08:14:45.5089056Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:45.5089505Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:45.5090248Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:45.5090879Z ##[endgroup] 2024-06-26T08:14:45.8166412Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:14:45.8167278Z With the provided path, there will be 1 file uploaded 2024-06-26T08:14:45.8167914Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:14:45.8193906Z Starting upload of test-reports-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142.zip 2024-06-26T08:14:46.0478092Z Finished upload of test-reports-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142.zip 2024-06-26T08:14:46.0611133Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:14:46.0611592Z with: 2024-06-26T08:14:46.0611872Z s3-bucket: gha-artifacts 2024-06-26T08:14:46.0612298Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:14:46.0612763Z retention-days: 14 2024-06-26T08:14:46.0613095Z if-no-files-found: ignore 2024-06-26T08:14:46.0613468Z path: logs-*.zip 2024-06-26T08:14:46.0613773Z name: artifact 2024-06-26T08:14:46.0614066Z region: us-east-1 2024-06-26T08:14:46.0614362Z env: 2024-06-26T08:14:46.0614639Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:46.0615108Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:46.0615832Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:46.0616463Z ##[endgroup] 2024-06-26T08:14:46.3660819Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:14:46.3661454Z With the provided path, there will be 1 file uploaded 2024-06-26T08:14:46.3662079Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:14:46.3687599Z Starting upload of logs-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142.zip 2024-06-26T08:14:46.6130092Z Finished upload of logs-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911142.zip 2024-06-26T08:14:46.6263005Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:14:46.6263459Z with: 2024-06-26T08:14:46.6263739Z s3-bucket: gha-artifacts 2024-06-26T08:14:46.6264164Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:14:46.6264807Z retention-days: 14 2024-06-26T08:14:46.6265136Z if-no-files-found: ignore 2024-06-26T08:14:46.6265506Z path: debug-*.zip 2024-06-26T08:14:46.6265820Z name: artifact 2024-06-26T08:14:46.6266114Z region: us-east-1 2024-06-26T08:14:46.6266413Z env: 2024-06-26T08:14:46.6266689Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:46.6267142Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:46.6267869Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:46.6268505Z ##[endgroup] 2024-06-26T08:14:46.9309000Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:14:46.9434963Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:14:46.9435448Z # shellcheck disable=SC2156 2024-06-26T08:14:46.9436389Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:14:46.9444368Z shell: /usr/bin/bash -e {0} 2024-06-26T08:14:46.9444717Z env: 2024-06-26T08:14:46.9444991Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:46.9445443Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:46.9446173Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:46.9446810Z ##[endgroup] 2024-06-26T08:14:47.1069594Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:14:47.1070161Z with: 2024-06-26T08:14:47.1070422Z env: 2024-06-26T08:14:47.1070704Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:47.1071166Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:47.1071901Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:47.1072543Z ##[endgroup] 2024-06-26T08:14:47.1089757Z ##[group]Run set -eou pipefail 2024-06-26T08:14:47.1090166Z set -eou pipefail 2024-06-26T08:14:47.1090519Z  2024-06-26T08:14:47.1091135Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:14:47.1091771Z for _ in $(seq 1440); do 2024-06-26T08:14:47.1092249Z  # Break if no ssh session exists anymore 2024-06-26T08:14:47.1092750Z  if [ "$(who)" = "" ]; then 2024-06-26T08:14:47.1093162Z  break 2024-06-26T08:14:47.1093540Z  fi 2024-06-26T08:14:47.1093853Z  echo "." 2024-06-26T08:14:47.1094182Z  sleep 5 2024-06-26T08:14:47.1094499Z done 2024-06-26T08:14:47.1101967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:47.1102455Z env: 2024-06-26T08:14:47.1102731Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:47.1103192Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:47.1103933Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:47.1104783Z ##[endgroup] 2024-06-26T08:14:47.1124764Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:14:47.1160593Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:14:47.1161372Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:14:47.1161971Z # shellcheck disable=SC2046 2024-06-26T08:14:47.1162448Z docker stop $(docker ps -q) || true 2024-06-26T08:14:47.1162930Z # Prune all of the docker images 2024-06-26T08:14:47.1163388Z docker system prune -af 2024-06-26T08:14:47.1170037Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:47.1170538Z env: 2024-06-26T08:14:47.1170807Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:14:47.1171391Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:14:47.1172132Z DOCKER_CONTAINER_ID: 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:47.1172770Z ##[endgroup] 2024-06-26T08:14:47.4342679Z 8bd3d1782b65 2024-06-26T08:14:48.1824103Z Deleted Containers: 2024-06-26T08:14:48.1824743Z 8bd3d1782b65d74e0b542551791381ddd8fe15fd6855349139acbdfe9bd641b5 2024-06-26T08:14:48.1825187Z 2024-06-26T08:14:53.2940701Z Deleted Images: 2024-06-26T08:14:53.2942460Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:14:53.2945312Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:14:53.2947350Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:14:53.2948525Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:14:53.2949937Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:14:53.2950958Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:14:53.2951822Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:14:53.2952665Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:14:53.2953501Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:14:53.2954447Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:14:53.2955270Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:14:53.2956070Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:14:53.2956864Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:14:53.2957695Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:14:53.2958513Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:14:53.2959356Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:14:53.2960447Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:14:53.2961374Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:14:53.2962306Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:14:53.2963218Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:14:53.2964137Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:14:53.2965063Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:14:53.2965963Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:14:53.2966878Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:14:53.2967773Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:14:53.2968686Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:14:53.2969616Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:14:53.2970835Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:14:53.2971816Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:14:53.2972653Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:14:53.2973484Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:14:53.2974288Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:14:53.2975092Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:14:53.2975898Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:14:53.2976726Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:14:53.2977547Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:14:53.2978366Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:14:53.2979186Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:14:53.2979996Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:14:53.2980808Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:14:53.2981617Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:14:53.2982515Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:14:53.2983337Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:14:53.2984150Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:14:53.2985093Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:14:53.2985913Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:14:53.2986862Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:14:53.2987755Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:14:53.2988579Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:14:53.2989393Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:14:53.2990200Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:14:53.2991033Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:14:53.2991869Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:14:53.2992698Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:14:53.2993513Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:14:53.2994336Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:14:53.2995160Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:14:53.2995973Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:14:53.2996771Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:14:53.2997616Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:14:53.2998469Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:14:53.2999288Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:14:53.3000119Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:14:53.3000943Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:14:53.3001772Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:14:53.3002599Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:14:53.3003393Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:14:53.3004196Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:14:53.3005019Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:14:53.3005840Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:14:53.3006653Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:14:53.3019001Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:14:53.3020086Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:14:53.3020926Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:14:53.3021778Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:14:53.3022601Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:14:53.3023449Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:14:53.3024298Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:14:53.3025346Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:14:53.3026316Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:14:53.3027136Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:14:53.3027687Z 2024-06-26T08:14:53.3027836Z Total reclaimed space: 36.35GB 2024-06-26T08:14:53.3090527Z Post job cleanup. 2024-06-26T08:14:53.3138085Z Post job cleanup. 2024-06-26T08:14:53.3971224Z [command]/usr/bin/git version 2024-06-26T08:14:53.4012822Z git version 2.40.1 2024-06-26T08:14:53.4045225Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/0c67dc1a-534e-4d71-ba97-d1e541f568aa/.gitconfig' 2024-06-26T08:14:53.4057396Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/0c67dc1a-534e-4d71-ba97-d1e541f568aa' before making global git config changes 2024-06-26T08:14:53.4058697Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:14:53.4062433Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:14:53.4089383Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:14:53.4112111Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:14:53.4320951Z Entering 'android/libs/fbjni' 2024-06-26T08:14:53.4356752Z Entering 'third_party/FP16' 2024-06-26T08:14:53.4392430Z Entering 'third_party/FXdiv' 2024-06-26T08:14:53.4429281Z Entering 'third_party/NNPACK' 2024-06-26T08:14:53.4464198Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:14:53.4498812Z Entering 'third_party/XNNPACK' 2024-06-26T08:14:53.4543985Z Entering 'third_party/benchmark' 2024-06-26T08:14:53.4580831Z Entering 'third_party/cpp-httplib' 2024-06-26T08:14:53.4616779Z Entering 'third_party/cpuinfo' 2024-06-26T08:14:53.4653195Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:14:53.4686764Z Entering 'third_party/cutlass' 2024-06-26T08:14:53.4727079Z Entering 'third_party/eigen' 2024-06-26T08:14:53.4765242Z Entering 'third_party/fbgemm' 2024-06-26T08:14:53.4799261Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:14:53.4832915Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:14:53.4868419Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:14:53.4908362Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:14:53.4941484Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:14:53.4975826Z Entering 'third_party/flatbuffers' 2024-06-26T08:14:53.5014946Z Entering 'third_party/fmt' 2024-06-26T08:14:53.5050201Z Entering 'third_party/foxi' 2024-06-26T08:14:53.5085090Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:14:53.5120439Z Entering 'third_party/gloo' 2024-06-26T08:14:53.5155878Z Entering 'third_party/googletest' 2024-06-26T08:14:53.5191305Z Entering 'third_party/ideep' 2024-06-26T08:14:53.5224916Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:14:53.5264236Z Entering 'third_party/ittapi' 2024-06-26T08:14:53.5298766Z Entering 'third_party/kineto' 2024-06-26T08:14:53.5333563Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:14:53.5368263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:14:53.5405507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:14:53.5438956Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:14:53.5473224Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:14:53.5506029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:14:53.5542277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:14:53.5577161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:14:53.5612259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:14:53.5646102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:14:53.5682361Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:14:53.5716580Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:14:53.5753155Z Entering 'third_party/mimalloc' 2024-06-26T08:14:53.5789305Z Entering 'third_party/nccl/nccl' 2024-06-26T08:14:53.5823817Z Entering 'third_party/nlohmann' 2024-06-26T08:14:53.5859405Z Entering 'third_party/onnx' 2024-06-26T08:14:53.5909420Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:14:53.5944172Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:14:53.5980938Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:14:53.6018930Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:14:53.6054647Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:14:53.6089994Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:14:53.6122484Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:14:53.6157872Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:14:53.6192194Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:14:53.6226636Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:14:53.6259613Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:14:53.6296697Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:14:53.6334148Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:14:53.6382479Z Entering 'third_party/pocketfft' 2024-06-26T08:14:53.6417713Z Entering 'third_party/protobuf' 2024-06-26T08:14:53.6455436Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:14:53.6490130Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:14:53.6527088Z Entering 'third_party/psimd' 2024-06-26T08:14:53.6562103Z Entering 'third_party/pthreadpool' 2024-06-26T08:14:53.6596635Z Entering 'third_party/pybind11' 2024-06-26T08:14:53.6631745Z Entering 'third_party/python-peachpy' 2024-06-26T08:14:53.6668748Z Entering 'third_party/sleef' 2024-06-26T08:14:53.6701711Z Entering 'third_party/tensorpipe' 2024-06-26T08:14:53.6735277Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:14:53.6769717Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:14:53.6802138Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:14:53.6836570Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:14:53.6870633Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:14:53.6920280Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:14:53.6941655Z http.https://github.com/.extraheader 2024-06-26T08:14:53.6951014Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:14:53.6974376Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:14:53.7183796Z Entering 'android/libs/fbjni' 2024-06-26T08:14:53.7204074Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7227846Z Entering 'third_party/FP16' 2024-06-26T08:14:53.7249929Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7272842Z Entering 'third_party/FXdiv' 2024-06-26T08:14:53.7293252Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7313823Z Entering 'third_party/NNPACK' 2024-06-26T08:14:53.7335400Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7357511Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:14:53.7377758Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7401869Z Entering 'third_party/XNNPACK' 2024-06-26T08:14:53.7423661Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7458783Z Entering 'third_party/benchmark' 2024-06-26T08:14:53.7480564Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7503196Z Entering 'third_party/cpp-httplib' 2024-06-26T08:14:53.7524738Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7547764Z Entering 'third_party/cpuinfo' 2024-06-26T08:14:53.7568981Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7593300Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:14:53.7614934Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7637566Z Entering 'third_party/cutlass' 2024-06-26T08:14:53.7659408Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7688304Z Entering 'third_party/eigen' 2024-06-26T08:14:53.7710715Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7734781Z Entering 'third_party/fbgemm' 2024-06-26T08:14:53.7756617Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7779964Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:14:53.7801139Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7824313Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:14:53.7845787Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7868713Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:14:53.7889390Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7916842Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:14:53.7937874Z http.https://github.com/.extraheader 2024-06-26T08:14:53.7961488Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:14:53.7983088Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8007882Z Entering 'third_party/flatbuffers' 2024-06-26T08:14:53.8030625Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8055285Z Entering 'third_party/fmt' 2024-06-26T08:14:53.8076920Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8099915Z Entering 'third_party/foxi' 2024-06-26T08:14:53.8121192Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8143390Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:14:53.8165320Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8188327Z Entering 'third_party/gloo' 2024-06-26T08:14:53.8210346Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8233604Z Entering 'third_party/googletest' 2024-06-26T08:14:53.8255453Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8278467Z Entering 'third_party/ideep' 2024-06-26T08:14:53.8300165Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8321696Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:14:53.8342441Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8371833Z Entering 'third_party/ittapi' 2024-06-26T08:14:53.8392504Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8416813Z Entering 'third_party/kineto' 2024-06-26T08:14:53.8439166Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8461484Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:14:53.8482575Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8505744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:14:53.8526299Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8550924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:14:53.8572310Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8595398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:14:53.8615882Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8639075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:14:53.8660221Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8682456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:14:53.8703381Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8728219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:14:53.8748551Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8771726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:14:53.8794241Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8817416Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:14:53.8838316Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8863268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:14:53.8884280Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8909306Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:14:53.8930158Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8953298Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:14:53.8975053Z http.https://github.com/.extraheader 2024-06-26T08:14:53.8998557Z Entering 'third_party/mimalloc' 2024-06-26T08:14:53.9021581Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9045945Z Entering 'third_party/nccl/nccl' 2024-06-26T08:14:53.9067493Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9090008Z Entering 'third_party/nlohmann' 2024-06-26T08:14:53.9113399Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9136638Z Entering 'third_party/onnx' 2024-06-26T08:14:53.9158410Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9195365Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:14:53.9217615Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9240223Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:14:53.9261267Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9286009Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:14:53.9307634Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9332150Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:14:53.9352667Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9376792Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:14:53.9397737Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9421056Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:14:53.9442525Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9464898Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:14:53.9485678Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9510753Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:14:53.9531334Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9555483Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:14:53.9576950Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9600155Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:14:53.9620756Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9642751Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:14:53.9663145Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9688586Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:14:53.9710501Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9733712Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:14:53.9755275Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9795231Z Entering 'third_party/pocketfft' 2024-06-26T08:14:53.9817875Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9842960Z Entering 'third_party/protobuf' 2024-06-26T08:14:53.9863214Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9886362Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:14:53.9907487Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9930625Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:14:53.9952040Z http.https://github.com/.extraheader 2024-06-26T08:14:53.9976704Z Entering 'third_party/psimd' 2024-06-26T08:14:53.9998286Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0021102Z Entering 'third_party/pthreadpool' 2024-06-26T08:14:54.0042282Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0066123Z Entering 'third_party/pybind11' 2024-06-26T08:14:54.0086996Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0109968Z Entering 'third_party/python-peachpy' 2024-06-26T08:14:54.0131217Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0153982Z Entering 'third_party/sleef' 2024-06-26T08:14:54.0176295Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0200253Z Entering 'third_party/tensorpipe' 2024-06-26T08:14:54.0221460Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0245145Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:14:54.0266766Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0290652Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:14:54.0312035Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0335211Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:14:54.0356943Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0380434Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:14:54.0402494Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0425287Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:14:54.0446942Z http.https://github.com/.extraheader 2024-06-26T08:14:54.0552017Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:14:54.0569207Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:14:54.0575968Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:14:54.0576493Z ##[endgroup] 2024-06-26T08:14:54.9300189Z Cleaning up orphan processes