2024-08-20T22:09:46.9793125Z Current runner version: '2.319.1' 2024-08-20T22:09:46.9799244Z Runner name: 'i-01499a1e34cc6562b' 2024-08-20T22:09:46.9800055Z Runner group name: 'Default' 2024-08-20T22:09:46.9800924Z Machine name: 'ip-10-0-17-178' 2024-08-20T22:09:46.9817955Z Testing runner upgrade compatibility 2024-08-20T22:09:47.1755480Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:09:47.1757524Z Actions: read 2024-08-20T22:09:47.1758045Z Attestations: read 2024-08-20T22:09:47.1758567Z Checks: read 2024-08-20T22:09:47.1759135Z Contents: read 2024-08-20T22:09:47.1759663Z Deployments: read 2024-08-20T22:09:47.1760150Z Discussions: read 2024-08-20T22:09:47.1760726Z Issues: read 2024-08-20T22:09:47.1761234Z Metadata: read 2024-08-20T22:09:47.1761710Z Packages: read 2024-08-20T22:09:47.1762282Z Pages: read 2024-08-20T22:09:47.1762767Z PullRequests: read 2024-08-20T22:09:47.1763427Z RepositoryProjects: read 2024-08-20T22:09:47.1764014Z SecurityEvents: read 2024-08-20T22:09:47.1764558Z Statuses: read 2024-08-20T22:09:47.1765308Z ##[endgroup] 2024-08-20T22:09:47.1768264Z Secret source: Actions 2024-08-20T22:09:47.1769054Z Prepare workflow directory 2024-08-20T22:09:47.2680537Z Prepare all required actions 2024-08-20T22:09:47.2845429Z Getting action download info 2024-08-20T22:09:47.4617996Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:09:47.9620454Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:09:51.9522212Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:09:52.1342740Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:09:52.5777752Z Getting action download info 2024-08-20T22:09:52.6725902Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:09:52.8834880Z Getting action download info 2024-08-20T22:09:52.9668669Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:09:53.1292713Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:09:53.1295074Z ##[group] Inputs 2024-08-20T22:09:53.1295569Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:09:53.1304871Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:09:53.1314992Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:09:53.1316167Z sync-tag: 2024-08-20T22:09:53.1317344Z timeout-minutes: 240 2024-08-20T22:09:53.1317697Z use-gha: 2024-08-20T22:09:53.1318001Z dashboard-tag: 2024-08-20T22:09:53.1318335Z s3-bucket: gha-artifacts 2024-08-20T22:09:53.1318696Z aws-role-to-assume: 2024-08-20T22:09:53.1319039Z ##[endgroup] 2024-08-20T22:09:53.1320065Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:09:53.2056408Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:09:53.2202588Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:09:53.2214604Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:09:53.2215129Z ##[endgroup] 2024-08-20T22:09:54.3572619Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:09:54.3573179Z Instance Type: g5.4xlarge 2024-08-20T22:09:54.3573877Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:09:54.3574427Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:00.1042265Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:00.1042831Z with: 2024-08-20T22:10:00.1043570Z github-secret: *** 2024-08-20T22:10:00.1044519Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:00.1045513Z activate-with-label: false 2024-08-20T22:10:00.1045885Z label: with-ssh 2024-08-20T22:10:00.1046220Z remove-existing-keys: true 2024-08-20T22:10:00.1046586Z fail-silently: true 2024-08-20T22:10:00.1046912Z env: 2024-08-20T22:10:00.1047198Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.1047550Z ##[endgroup] 2024-08-20T22:10:00.1931242Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:00.1934140Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:00.4704171Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:00.5475061Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:00.5475666Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:00.6392707Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:00.6407281Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:00.6432328Z Login using: ssh ec2-user@ec2-3-234-182-193.compute-1.amazonaws.com 2024-08-20T22:10:00.6433112Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:00.6433920Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:00.6562664Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:00.6563230Z with: 2024-08-20T22:10:00.6563517Z submodules: recursive 2024-08-20T22:10:00.6563857Z fetch-depth: 0 2024-08-20T22:10:00.6564160Z env: 2024-08-20T22:10:00.6564443Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.6564790Z ##[endgroup] 2024-08-20T22:10:00.6762808Z ##[group]Run retry () { 2024-08-20T22:10:00.6763189Z retry () { 2024-08-20T22:10:00.6763894Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:00.6764454Z } 2024-08-20T22:10:00.6764767Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.6765197Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:00.6765680Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.6766135Z else 2024-08-20T22:10:00.6766482Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.6766916Z fi 2024-08-20T22:10:00.6767258Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.6780566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:00.6781081Z env: 2024-08-20T22:10:00.6781358Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.6781697Z NO_SUDO: 2024-08-20T22:10:00.6781971Z ##[endgroup] 2024-08-20T22:10:00.6814107Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:00.8061480Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:00.8061924Z with: 2024-08-20T22:10:00.8062252Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:00.8062689Z fetch-depth: 0 2024-08-20T22:10:00.8063002Z submodules: recursive 2024-08-20T22:10:00.8063331Z quiet-checkout: true 2024-08-20T22:10:00.8063675Z repository: pytorch/pytorch 2024-08-20T22:10:00.8064148Z token: *** 2024-08-20T22:10:00.8064435Z ssh-strict: true 2024-08-20T22:10:00.8064752Z persist-credentials: true 2024-08-20T22:10:00.8065103Z clean: true 2024-08-20T22:10:00.8065431Z sparse-checkout-cone-mode: true 2024-08-20T22:10:00.8065817Z lfs: false 2024-08-20T22:10:00.8066108Z set-safe-directory: true 2024-08-20T22:10:00.8066437Z env: 2024-08-20T22:10:00.8066703Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.8067031Z ##[endgroup] 2024-08-20T22:10:00.9018337Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:00.9019875Z ##[group]Getting Git version info 2024-08-20T22:10:00.9020613Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:00.9021629Z [command]/usr/bin/git version 2024-08-20T22:10:00.9021986Z git version 2.40.1 2024-08-20T22:10:00.9023440Z ##[endgroup] 2024-08-20T22:10:00.9034622Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/274ff4bf-da1d-44b0-b93b-71d0508f5544' before making global git config changes 2024-08-20T22:10:00.9035840Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:00.9037374Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:00.9090691Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:00.9094353Z ##[group]Initializing the repository 2024-08-20T22:10:00.9097239Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:00.9140515Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:00.9141514Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:00.9142330Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:00.9143013Z hint: 2024-08-20T22:10:00.9143472Z hint: git config --global init.defaultBranch 2024-08-20T22:10:00.9143934Z hint: 2024-08-20T22:10:00.9144462Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:00.9145297Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:00.9146126Z hint: 2024-08-20T22:10:00.9146456Z hint: git branch -m 2024-08-20T22:10:00.9148951Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:00.9154924Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:00.9197974Z ##[endgroup] 2024-08-20T22:10:00.9199127Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:00.9200278Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:00.9240707Z ##[endgroup] 2024-08-20T22:10:00.9241255Z ##[group]Setting up auth 2024-08-20T22:10:00.9246808Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:00.9287702Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:00.9679124Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:00.9721856Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:01.0078875Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:01.0133305Z ##[endgroup] 2024-08-20T22:10:01.0133875Z ##[group]Fetching the repository 2024-08-20T22:10:01.0139307Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:04.3477616Z remote: Enumerating objects: 1008509 2024-08-20T22:10:04.3478162Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:04.3482303Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:04.3483134Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:04.3483967Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:04.3484762Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:04.3485520Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:04.3486164Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:04.3486674Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:04.3487176Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:04.3487693Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:04.3488192Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:04.3488689Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:04.3489183Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:04.3489682Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:04.3490287Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:04.3490821Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:04.3491391Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:04.3491889Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:04.3492403Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:04.3492959Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:04.3493450Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:04.3493975Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:04.3494473Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:04.3495069Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:04.3495704Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:04.3496242Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:04.3496911Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:04.3497557Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:04.3498080Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:04.3498934Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:04.3499443Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:04.3499941Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:04.3500439Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:04.3500931Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:04.3501430Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:04.3502073Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:04.3502573Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:04.3503067Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:04.3503571Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:04.3504068Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:04.3504560Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:04.3505057Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:04.3505566Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:04.3506067Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:04.3506562Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:04.3507062Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:04.3507561Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:04.3508060Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:04.3508658Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:04.3509156Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:04.3509654Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:04.3510151Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:04.3510643Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:04.3511144Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:04.3511642Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:04.3512146Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:04.3512635Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:04.3513134Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:04.3513632Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:04.3514125Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:04.3514627Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:04.3515162Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:04.3515693Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:04.3516217Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:04.3516748Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:04.3517619Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:04.3518129Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:04.3518645Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:04.3519152Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:04.3519658Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:04.3520166Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:04.3520670Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:04.3521195Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:04.3521696Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:04.3522214Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:04.3522721Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:04.3523226Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:04.3523726Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:04.3524234Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:04.3524735Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:04.3525382Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:04.3525886Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:04.3526392Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:04.3526902Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:04.3527406Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:04.3527903Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:04.3528406Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:04.3529028Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:04.3529536Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:04.3530036Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:04.3530546Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:04.3531047Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:04.3531554Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:04.3532063Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:04.3532565Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:04.3533071Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:04.3533583Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:04.3534082Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:04.3534591Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:04.3535101Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:04.3535605Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:04.3536104Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:04.3536652Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:04.3655424Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:04.3910310Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:04.4191525Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:04.4628107Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:04.5438123Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:04.5900488Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:04.6524507Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:04.6866064Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:04.7258108Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:04.7473900Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:04.7749616Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:04.7981814Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:04.8157706Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:04.8345076Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:04.8470598Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:04.8581283Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:04.8685422Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:04.8754844Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:04.8807554Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:04.8854494Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:04.8885828Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:04.8904964Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:04.8909230Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:04.8914105Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:04.8922125Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:04.8926681Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:04.8937044Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:04.8942595Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:04.8962246Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:04.8975326Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:04.8984665Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:04.8995025Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:04.8999719Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:04.9010216Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:04.9017789Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:04.9021609Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:04.9029565Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:04.9035741Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:04.9044771Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:04.9051645Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:04.9057194Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:04.9062851Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:04.9071559Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:04.9075396Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:04.9083067Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:04.9087996Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:04.9092862Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:04.9096569Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:04.9099661Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:04.9103113Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:04.9105698Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:04.9109135Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:04.9113171Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:04.9114114Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:04.9115609Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:04.9116652Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:04.9120246Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:04.9121174Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:04.9121858Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:04.9122542Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:04.9123242Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:04.9123916Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:04.9124563Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:04.9125213Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:04.9125879Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:04.9131676Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:04.9137769Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:04.9142405Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:04.9145976Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:04.9151128Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:04.9156007Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:04.9159096Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:04.9162186Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:04.9165987Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:04.9168928Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:04.9172410Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:04.9174568Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:04.9177855Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:04.9180162Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:04.9181491Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:04.9183982Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:04.9186366Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:04.9187867Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:04.9190477Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:04.9191564Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:04.9193686Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:04.9195629Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:04.9196672Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:04.9199128Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:04.9201034Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:04.9202377Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:04.9203620Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:04.9204901Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:04.9205570Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:04.9206370Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:04.9208277Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:04.9210106Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:04.9211154Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:04.9212378Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:04.9213803Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:04.9214477Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:04.9215093Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:28.5339839Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:10:50.9682359Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:10:50.9720696Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:50.9725851Z ##[endgroup] 2024-08-20T22:10:50.9726583Z ##[group]Determining the checkout info 2024-08-20T22:10:50.9727406Z ##[endgroup] 2024-08-20T22:10:50.9730203Z ##[group]Checking out the ref 2024-08-20T22:10:50.9731338Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:52.6821095Z ##[endgroup] 2024-08-20T22:10:52.6821778Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:10:52.6823800Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:52.6878713Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:10:52.6918267Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:10:52.6956829Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:10:52.6993153Z ##[endgroup] 2024-08-20T22:10:52.6993770Z ##[group]Fetching submodules 2024-08-20T22:10:52.6997480Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:10:52.7375697Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:10:52.7743930Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:10:52.7745388Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:10:52.7748436Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:10:52.7751989Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:10:52.7755928Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:52.7758670Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:10:52.7762283Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:10:52.7765820Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:10:52.7769347Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:10:52.7773636Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:10:52.7777688Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:10:52.7781928Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:10:52.7786263Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:10:52.7791062Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:10:52.7795482Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:10:52.7800209Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:52.7808441Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:10:52.7813159Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:10:52.7818251Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:10:52.7823402Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:10:52.7828258Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:10:52.7833568Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:10:52.7838482Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:10:52.7843772Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:10:52.7848746Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:10:52.7854479Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:10:52.7859625Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:10:52.7865118Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:10:52.7870747Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:10:52.7876445Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:10:52.7882075Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:10:52.7888028Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:10:52.7893637Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:10:52.7903022Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:10:52.7938675Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:10:53.0905926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:10:53.2728951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:10:53.4750884Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:10:53.7163934Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:10:55.6515624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:06.9890781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:07.3999016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:07.8754247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:08.4885710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:09.7476771Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:11.8598004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:17.7355188Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:19.1563796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:20.9734987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:22.2880701Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:22.7388756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:23.0815886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:24.2426407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:24.6781406Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:24.9192377Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:26.4925982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:27.1697257Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:27.8413761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:33.9072635Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:35.8957799Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:40.1256585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:40.3522323Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:11:49.3008683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:11:49.4756018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:11:49.6820180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:11:50.5653768Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:11:50.8618419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:11:51.4912962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:11:51.9336917Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:11:51.9481825Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:11:51.9597048Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:11:51.9904513Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:11:52.0331788Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:11:53.1826902Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:11:53.2102747Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:11:53.2617329Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:11:53.3741325Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:11:53.4144153Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:11:54.0344030Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:11:54.3199547Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:11:54.4138551Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:11:54.4160596Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:11:54.4164146Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:11:54.4167592Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:11:54.4171109Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:11:54.4174856Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:11:54.4206698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:11:55.3625130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:11:55.9953638Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:11:58.2108302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:11:59.2779754Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:11:59.6073578Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:11:59.7172046Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:00.2213529Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:00.2916552Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:00.3062648Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:00.4566407Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:00.5026569Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:00.5473831Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:00.5797574Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:00.6328657Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:00.6485218Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:00.6505333Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:00.6542478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:14.0442958Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:14.0660014Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:14.1640698Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:14.1662801Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:14.1665919Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:14.1669555Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:14.1699582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:14.6700623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:15.8617574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:17.0234027Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:17.0254396Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:17.0257639Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:17.0261247Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:17.0265451Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:17.0268629Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:17.0272725Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:17.0276526Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:17.0280394Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:17.0312364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:17.8824042Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:18.2589171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:19.4514729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:19.7211581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:20.2486943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:21.2774410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:28.7871510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:29.2541885Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:29.2782986Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:29.3256180Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:29.3424148Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:29.3444605Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:29.3476195Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:29.6330796Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:29.6552766Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:29.7027337Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:29.8242798Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:29.8445621Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:29.8897176Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:29.9547371Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:30.0003090Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:30.0302950Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:30.1551942Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:30.6644139Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:30.6683109Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:30.6686022Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:30.6718444Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:31.1037671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:32.0643382Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:32.1059288Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:32.1925800Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:32.1948843Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:32.1952247Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:32.1955497Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:32.1958915Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:32.1962666Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:32.1966182Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:32.1969723Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:32.1973106Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:32.2008412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:32.6306841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:33.6850565Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:33.9801620Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:40.0854261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:40.3575895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:40.6040807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:40.8913490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:12:47.5941899Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:12:47.6411688Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:12:47.6601970Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:12:47.7820346Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:12:47.7983740Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:12:47.8167646Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:12:47.8372324Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:12:47.8391204Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:47.8394578Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:47.8425893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:12:49.7706916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:12:51.0539722Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:12:51.1072879Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:51.7717905Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:12:51.7862425Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:12:52.1011787Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:12:52.1037535Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:52.1040617Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:52.1071839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:12:52.5477147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:12:53.6082036Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:12:53.6881102Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:12:53.6997377Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:12:53.7155876Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:12:53.7600368Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:12:53.7941583Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:12:53.8418689Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:12:53.8761167Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:12:53.8783136Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:53.8786317Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:53.8789981Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:53.8793517Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:53.8824762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:12:54.9255707Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:12:55.1544269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:12:56.3046330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:12:57.2598370Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:12:57.2791061Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:12:57.3579197Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:12:57.3916050Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:12:57.3936208Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:57.3966839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:12:57.5996860Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:12:57.6044780Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:12:57.6407526Z Entering 'android/libs/fbjni' 2024-08-20T22:12:57.6460218Z Entering 'third_party/FP16' 2024-08-20T22:12:57.6512511Z Entering 'third_party/FXdiv' 2024-08-20T22:12:57.6565193Z Entering 'third_party/NNPACK' 2024-08-20T22:12:57.6618222Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:57.6670759Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:57.6739218Z Entering 'third_party/benchmark' 2024-08-20T22:12:57.6791214Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:57.6843419Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:57.6894699Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:57.6946979Z Entering 'third_party/cutlass' 2024-08-20T22:12:57.7006225Z Entering 'third_party/eigen' 2024-08-20T22:12:57.7060391Z Entering 'third_party/fbgemm' 2024-08-20T22:12:57.7110398Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:57.7161473Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:57.7215125Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:57.7277573Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:57.7328253Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:57.7381325Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:57.7436739Z Entering 'third_party/fmt' 2024-08-20T22:12:57.7489185Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:57.7542201Z Entering 'third_party/gloo' 2024-08-20T22:12:57.7594314Z Entering 'third_party/googletest' 2024-08-20T22:12:57.7647168Z Entering 'third_party/ideep' 2024-08-20T22:12:57.7695913Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:57.7760614Z Entering 'third_party/ittapi' 2024-08-20T22:12:57.7811452Z Entering 'third_party/kineto' 2024-08-20T22:12:57.7861794Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:57.7911372Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:57.7963000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:57.8012551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:57.8062696Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:57.8110573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:57.8165686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:57.8216347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:57.8270136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:57.8320489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:57.8373411Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:57.8426965Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:57.8480471Z Entering 'third_party/mimalloc' 2024-08-20T22:12:57.8533389Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:57.8585739Z Entering 'third_party/nlohmann' 2024-08-20T22:12:57.8643714Z Entering 'third_party/onnx' 2024-08-20T22:12:57.8711590Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:57.8763889Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:57.8820045Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:57.8874072Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:57.8923809Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:57.8974295Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:57.9025031Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:57.9076217Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:57.9126464Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:57.9175974Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:57.9224323Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:57.9277776Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:57.9331543Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:57.9404381Z Entering 'third_party/pocketfft' 2024-08-20T22:12:57.9458753Z Entering 'third_party/protobuf' 2024-08-20T22:12:57.9511909Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:57.9562510Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:57.9618037Z Entering 'third_party/psimd' 2024-08-20T22:12:57.9670703Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:57.9725247Z Entering 'third_party/pybind11' 2024-08-20T22:12:57.9777521Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:57.9830188Z Entering 'third_party/sleef' 2024-08-20T22:12:57.9882172Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:57.9934081Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:57.9983701Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:58.0033754Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:58.0083896Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:58.0132692Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:58.0206885Z ##[endgroup] 2024-08-20T22:12:58.0207539Z ##[group]Persisting credentials for submodules 2024-08-20T22:12:58.0209546Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:12:58.0573342Z Entering 'android/libs/fbjni' 2024-08-20T22:12:58.0640170Z Entering 'third_party/FP16' 2024-08-20T22:12:58.0707077Z Entering 'third_party/FXdiv' 2024-08-20T22:12:58.0775429Z Entering 'third_party/NNPACK' 2024-08-20T22:12:58.0846007Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:58.0912061Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:58.0995083Z Entering 'third_party/benchmark' 2024-08-20T22:12:58.1061916Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:58.1131305Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:58.1198539Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:58.1271223Z Entering 'third_party/cutlass' 2024-08-20T22:12:58.1348061Z Entering 'third_party/eigen' 2024-08-20T22:12:58.1416486Z Entering 'third_party/fbgemm' 2024-08-20T22:12:58.1482102Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:58.1548487Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:58.1613909Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:58.1686093Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:58.1752743Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:58.1822991Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:58.1896782Z Entering 'third_party/fmt' 2024-08-20T22:12:58.1964009Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:58.2031456Z Entering 'third_party/gloo' 2024-08-20T22:12:58.2098460Z Entering 'third_party/googletest' 2024-08-20T22:12:58.2165111Z Entering 'third_party/ideep' 2024-08-20T22:12:58.2229655Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:58.2305500Z Entering 'third_party/ittapi' 2024-08-20T22:12:58.2373347Z Entering 'third_party/kineto' 2024-08-20T22:12:58.2438902Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:58.2503806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:58.2572537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:58.2639994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:58.2712485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:58.2777683Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:58.2848589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:58.2916836Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:58.2984164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:58.3051900Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:58.3125976Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:58.3192709Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:58.3262547Z Entering 'third_party/mimalloc' 2024-08-20T22:12:58.3330295Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:58.3397091Z Entering 'third_party/nlohmann' 2024-08-20T22:12:58.3465618Z Entering 'third_party/onnx' 2024-08-20T22:12:58.3547335Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:58.3619643Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:58.3691387Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:58.3762669Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:58.3829365Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:58.3895250Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:58.3961661Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:58.4028927Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:58.4094378Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:58.4159751Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:58.4223004Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:58.4290870Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:58.4361301Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:58.4452873Z Entering 'third_party/pocketfft' 2024-08-20T22:12:58.4520037Z Entering 'third_party/protobuf' 2024-08-20T22:12:58.4587459Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:58.4652966Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:58.4722281Z Entering 'third_party/psimd' 2024-08-20T22:12:58.4788303Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:58.4859887Z Entering 'third_party/pybind11' 2024-08-20T22:12:58.4926817Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:58.4992863Z Entering 'third_party/sleef' 2024-08-20T22:12:58.5060971Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:58.5129547Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:58.5195048Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:58.5260820Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:58.5328057Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:58.5392043Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:58.5490802Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:12:58.5850179Z Entering 'android/libs/fbjni' 2024-08-20T22:12:58.5916567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:12:58.5938820Z Entering 'third_party/FP16' 2024-08-20T22:12:58.6001410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:12:58.6023032Z Entering 'third_party/FXdiv' 2024-08-20T22:12:58.6084756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:12:58.6105697Z Entering 'third_party/NNPACK' 2024-08-20T22:12:58.6167698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:12:58.6189910Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:58.6251844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:12:58.6273293Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:58.6337304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:12:58.6375497Z Entering 'third_party/benchmark' 2024-08-20T22:12:58.6437350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:58.6458730Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:58.6521206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:12:58.6542369Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:58.6604462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:12:58.6626853Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:58.6688495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:12:58.6709529Z Entering 'third_party/cutlass' 2024-08-20T22:12:58.6771251Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:12:58.6800746Z Entering 'third_party/eigen' 2024-08-20T22:12:58.6862223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:12:58.6885773Z Entering 'third_party/fbgemm' 2024-08-20T22:12:58.6950040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:12:58.6968901Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:58.7030641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:12:58.7051377Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:58.7116705Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:12:58.7138269Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:58.7197758Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:12:58.7224560Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:58.7284127Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.7305080Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:58.7369862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:12:58.7393456Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:58.7455795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:12:58.7478994Z Entering 'third_party/fmt' 2024-08-20T22:12:58.7540177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:12:58.7561025Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:58.7623595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:12:58.7643628Z Entering 'third_party/gloo' 2024-08-20T22:12:58.7704271Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:12:58.7726150Z Entering 'third_party/googletest' 2024-08-20T22:12:58.7786462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.7807392Z Entering 'third_party/ideep' 2024-08-20T22:12:58.7868581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:12:58.7886717Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:58.7947386Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:12:58.7978136Z Entering 'third_party/ittapi' 2024-08-20T22:12:58.8040764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:12:58.8062070Z Entering 'third_party/kineto' 2024-08-20T22:12:58.8128071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:12:58.8147737Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:58.8214357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:12:58.8234960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:58.8296463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:12:58.8318114Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:58.8380152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:12:58.8400667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:58.8463270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:12:58.8483471Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:58.8546601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:12:58.8564829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:58.8627862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:12:58.8652354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:58.8713908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:12:58.8734394Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:58.8796025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.8816547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:58.8878495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:12:58.8899590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:58.8962641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:12:58.8985579Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:58.9046849Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:12:58.9067599Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:58.9129331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.9155183Z Entering 'third_party/mimalloc' 2024-08-20T22:12:58.9217746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:12:58.9240962Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:58.9301531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:12:58.9323166Z Entering 'third_party/nlohmann' 2024-08-20T22:12:58.9383791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:12:58.9405895Z Entering 'third_party/onnx' 2024-08-20T22:12:58.9468813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:12:58.9502695Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:58.9564126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:58.9585544Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:58.9647964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:12:58.9674668Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:58.9737260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:12:58.9758557Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:58.9819296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:58.9843153Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:58.9903765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.9926677Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:58.9986859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:12:59.0006704Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:59.0069212Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:12:59.0090615Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:59.0150776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:12:59.0170504Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:59.0231979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:12:59.0251993Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:59.0316130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:12:59.0334432Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:59.0396033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:12:59.0417985Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:59.0479536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:12:59.0502368Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:59.0563561Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:12:59.0604744Z Entering 'third_party/pocketfft' 2024-08-20T22:12:59.0666304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:12:59.0687603Z Entering 'third_party/protobuf' 2024-08-20T22:12:59.0749334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:12:59.0770839Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:59.0835686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:59.0855320Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:59.0916705Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:59.0940994Z Entering 'third_party/psimd' 2024-08-20T22:12:59.1002804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:12:59.1024249Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:59.1086542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:12:59.1107472Z Entering 'third_party/pybind11' 2024-08-20T22:12:59.1169359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:12:59.1192554Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:59.1254781Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:12:59.1276120Z Entering 'third_party/sleef' 2024-08-20T22:12:59.1338279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:12:59.1359778Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:59.1422032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:12:59.1441860Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:59.1502432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:59.1523505Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:59.1583669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:12:59.1603429Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:59.1664123Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:12:59.1684811Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:59.1746068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:12:59.1764028Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:59.1826010Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:12:59.2513977Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:12:59.2876461Z Entering 'android/libs/fbjni' 2024-08-20T22:12:59.2933482Z Entering 'third_party/FP16' 2024-08-20T22:12:59.2985305Z Entering 'third_party/FXdiv' 2024-08-20T22:12:59.3036723Z Entering 'third_party/NNPACK' 2024-08-20T22:12:59.3092067Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:59.3145218Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:59.3212902Z Entering 'third_party/benchmark' 2024-08-20T22:12:59.3265481Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:59.3317428Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:59.3369572Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:59.3421892Z Entering 'third_party/cutlass' 2024-08-20T22:12:59.3481455Z Entering 'third_party/eigen' 2024-08-20T22:12:59.3535563Z Entering 'third_party/fbgemm' 2024-08-20T22:12:59.3590758Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:59.3643366Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:59.3693267Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:59.3750805Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:59.3799126Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:59.3853904Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:59.3915619Z Entering 'third_party/fmt' 2024-08-20T22:12:59.3967401Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:59.4019495Z Entering 'third_party/gloo' 2024-08-20T22:12:59.4070967Z Entering 'third_party/googletest' 2024-08-20T22:12:59.4125857Z Entering 'third_party/ideep' 2024-08-20T22:12:59.4175913Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:59.4236280Z Entering 'third_party/ittapi' 2024-08-20T22:12:59.4287102Z Entering 'third_party/kineto' 2024-08-20T22:12:59.4337474Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:59.4387629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:59.4446924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:59.4498482Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:59.4549942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:59.4599146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:59.4655350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:59.4707972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:59.4760729Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:59.4815615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:59.4872453Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:59.4923328Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:59.4976616Z Entering 'third_party/mimalloc' 2024-08-20T22:12:59.5028632Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:59.5080664Z Entering 'third_party/nlohmann' 2024-08-20T22:12:59.5137431Z Entering 'third_party/onnx' 2024-08-20T22:12:59.5206609Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:59.5258686Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:59.5315424Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:59.5368656Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:59.5419240Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:59.5467246Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:59.5515760Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:59.5567041Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:59.5616010Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:59.5668542Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:59.5716346Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:59.5769245Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:59.5823854Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:59.5895503Z Entering 'third_party/pocketfft' 2024-08-20T22:12:59.5950389Z Entering 'third_party/protobuf' 2024-08-20T22:12:59.6003606Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:59.6055119Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:59.6109406Z Entering 'third_party/psimd' 2024-08-20T22:12:59.6161096Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:59.6212915Z Entering 'third_party/pybind11' 2024-08-20T22:12:59.6264425Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:59.6315935Z Entering 'third_party/sleef' 2024-08-20T22:12:59.6368639Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:59.6419815Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:59.6469607Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:59.6518480Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:59.6568787Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:59.6617083Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:59.6697683Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:12:59.7060298Z Entering 'android/libs/fbjni' 2024-08-20T22:12:59.7112061Z Entering 'third_party/FP16' 2024-08-20T22:12:59.7164622Z Entering 'third_party/FXdiv' 2024-08-20T22:12:59.7216599Z Entering 'third_party/NNPACK' 2024-08-20T22:12:59.7269638Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:59.7322274Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:59.7390179Z Entering 'third_party/benchmark' 2024-08-20T22:12:59.7442942Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:59.7494773Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:59.7549138Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:59.7601402Z Entering 'third_party/cutlass' 2024-08-20T22:12:59.7662248Z Entering 'third_party/eigen' 2024-08-20T22:12:59.7716576Z Entering 'third_party/fbgemm' 2024-08-20T22:12:59.7768417Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:59.7824940Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:59.7874992Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:59.7932207Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:59.7982789Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:59.8037161Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:59.8091613Z Entering 'third_party/fmt' 2024-08-20T22:12:59.8143929Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:59.8195335Z Entering 'third_party/gloo' 2024-08-20T22:12:59.8247217Z Entering 'third_party/googletest' 2024-08-20T22:12:59.8298696Z Entering 'third_party/ideep' 2024-08-20T22:12:59.8348353Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:59.8412934Z Entering 'third_party/ittapi' 2024-08-20T22:12:59.8465247Z Entering 'third_party/kineto' 2024-08-20T22:12:59.8517185Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:59.8567586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:59.8620251Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:59.8670794Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:59.8720763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:59.8791442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:59.8846021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:59.8896824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:59.8948577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:59.9000364Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:59.9054580Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:59.9105399Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:59.9162979Z Entering 'third_party/mimalloc' 2024-08-20T22:12:59.9216076Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:59.9268943Z Entering 'third_party/nlohmann' 2024-08-20T22:12:59.9321870Z Entering 'third_party/onnx' 2024-08-20T22:12:59.9387161Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:59.9443140Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:59.9499755Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:59.9554976Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:59.9604819Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:59.9656142Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:59.9706159Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:59.9758598Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:59.9808305Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:59.9863822Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:59.9912496Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:59.9965432Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:00.0019473Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:00.0090985Z Entering 'third_party/pocketfft' 2024-08-20T22:13:00.0143608Z Entering 'third_party/protobuf' 2024-08-20T22:13:00.0197025Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:00.0248787Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:00.0307183Z Entering 'third_party/psimd' 2024-08-20T22:13:00.0359331Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:00.0410444Z Entering 'third_party/pybind11' 2024-08-20T22:13:00.0462081Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:00.0513314Z Entering 'third_party/sleef' 2024-08-20T22:13:00.0564778Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:00.0615156Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:00.0665428Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:00.0715971Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:00.0766917Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:00.0815059Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:00.0889863Z ##[endgroup] 2024-08-20T22:13:00.0938958Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:00.0976474Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:00.1197925Z Prepare all required actions 2024-08-20T22:13:00.1198385Z Getting action download info 2024-08-20T22:13:00.2693583Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:00.2694004Z env: 2024-08-20T22:13:00.2694276Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:00.2694606Z ##[endgroup] 2024-08-20T22:13:00.2754580Z ##[group]Run set -euo pipefail 2024-08-20T22:13:00.2755029Z set -euo pipefail 2024-08-20T22:13:00.2755428Z function get_ec2_metadata() { 2024-08-20T22:13:00.2755969Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:00.2756857Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:00.2757623Z  category=$1 2024-08-20T22:13:00.2758167Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:00.2758805Z  runner_name_str=i-01499a1e34cc6562b 2024-08-20T22:13:00.2759308Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:00.2759806Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:00.2760386Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:00.2761080Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:00.2761688Z  else 2024-08-20T22:13:00.2762183Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:00.2762763Z  fi 2024-08-20T22:13:00.2763057Z } 2024-08-20T22:13:00.2763428Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:00.2764014Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:00.2764678Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:00.2765252Z echo "system info $(uname -a)" 2024-08-20T22:13:00.2775620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:00.2776112Z env: 2024-08-20T22:13:00.2776401Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:00.2776755Z ##[endgroup] 2024-08-20T22:13:00.2891543Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:00.2949748Z instance-id: i-01499a1e34cc6562b 2024-08-20T22:13:00.3011735Z instance-type: g5.4xlarge 2024-08-20T22:13:00.3026386Z system info Linux ip-10-0-17-178.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:00.3050298Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:00.3051228Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:00.3061485Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:00.3062165Z env: 2024-08-20T22:13:00.3062450Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:00.3062800Z ##[endgroup] 2024-08-20T22:13:00.3140721Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:00.3141342Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:00.3141891Z  echo "Docker daemon is running..."; 2024-08-20T22:13:00.3142352Z else 2024-08-20T22:13:00.3142850Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:00.3143438Z fi 2024-08-20T22:13:00.3152377Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:00.3152883Z env: 2024-08-20T22:13:00.3153170Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:00.3153523Z ##[endgroup] 2024-08-20T22:13:00.3243178Z Docker daemon is running... 2024-08-20T22:13:00.3291294Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:00.3291862Z with: 2024-08-20T22:13:00.3292152Z shell: bash 2024-08-20T22:13:00.3292451Z timeout_minutes: 5 2024-08-20T22:13:00.3292783Z max_attempts: 3 2024-08-20T22:13:00.3293109Z retry_wait_seconds: 30 2024-08-20T22:13:00.3296185Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:00.3299121Z polling_interval_seconds: 1 2024-08-20T22:13:00.3299511Z warning_on_retry: true 2024-08-20T22:13:00.3299865Z continue_on_error: false 2024-08-20T22:13:00.3300213Z env: 2024-08-20T22:13:00.3300503Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:00.3300865Z AWS_RETRY_MODE: standard 2024-08-20T22:13:00.3301219Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:00.3301572Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:00.3301959Z ##[endgroup] 2024-08-20T22:13:01.5051518Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:01.5052365Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:01.5053217Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:01.5053737Z 2024-08-20T22:13:01.5053858Z Login Succeeded 2024-08-20T22:13:02.3877027Z Command completed after 1 attempt(s). 2024-08-20T22:13:02.3941788Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:02.3942499Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:02.3943148Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:02.3953589Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.3954088Z env: 2024-08-20T22:13:02.3954373Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.3954733Z ##[endgroup] 2024-08-20T22:13:02.4049350Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:02.4050144Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:02.4050754Z # shellcheck disable=SC2046 2024-08-20T22:13:02.4051232Z docker stop $(docker ps -q) || true 2024-08-20T22:13:02.4051722Z # Prune all of the docker images 2024-08-20T22:13:02.4052197Z docker system prune -af 2024-08-20T22:13:02.4061142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.4061657Z env: 2024-08-20T22:13:02.4061945Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.4062308Z ##[endgroup] 2024-08-20T22:13:02.4379110Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:02.4379905Z See 'docker stop --help'. 2024-08-20T22:13:02.4380466Z 2024-08-20T22:13:02.4380691Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:02.4381070Z 2024-08-20T22:13:02.4381225Z Stop one or more running containers 2024-08-20T22:13:02.4815369Z Total reclaimed space: 0B 2024-08-20T22:13:02.4861855Z ##[group]Run set +e 2024-08-20T22:13:02.4862221Z set +e 2024-08-20T22:13:02.4862526Z set -x 2024-08-20T22:13:02.4862826Z  2024-08-20T22:13:02.4863148Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:02.4863930Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:02.4864985Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:02.4865728Z # one is returned at random 2024-08-20T22:13:02.4866272Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:02.4866785Z  2024-08-20T22:13:02.4867120Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:02.4867737Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:02.4868807Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:02.4869369Z  2024-08-20T22:13:02.4869699Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:02.4870252Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:02.4870762Z  exit 1 2024-08-20T22:13:02.4871075Z  fi 2024-08-20T22:13:02.4871364Z fi 2024-08-20T22:13:02.4871643Z  2024-08-20T22:13:02.4872002Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:02.4872510Z  # Clean up any old records first 2024-08-20T22:13:02.4873020Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:02.4873482Z fi 2024-08-20T22:13:02.4873760Z  2024-08-20T22:13:02.4874192Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:02.4874738Z cat /etc/hosts 2024-08-20T22:13:02.4883624Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.4884128Z env: 2024-08-20T22:13:02.4884422Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.4884779Z ##[endgroup] 2024-08-20T22:13:02.4918878Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:02.4925530Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:02.4926187Z ++ tail -n1 2024-08-20T22:13:02.5577203Z + RESOLVED_IP=18.160.10.28 2024-08-20T22:13:02.5577838Z + '[' -z 18.160.10.28 ']' 2024-08-20T22:13:02.5578279Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:02.5596393Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T22:13:02.5597410Z + sudo tee -a /etc/hosts 2024-08-20T22:13:02.8499178Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:02.8522826Z + cat /etc/hosts 2024-08-20T22:13:02.8535555Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:02.8547984Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:02.8548752Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:02.8700797Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:02.8701397Z with: 2024-08-20T22:13:02.8702492Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8703702Z docker-build-dir: .ci/docker 2024-08-20T22:13:02.8704086Z working-directory: . 2024-08-20T22:13:02.8704559Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.8705085Z force-push: false 2024-08-20T22:13:02.8705394Z env: 2024-08-20T22:13:02.8705675Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.8706017Z ##[endgroup] 2024-08-20T22:13:02.8724998Z ##[group]Run set -ex 2024-08-20T22:13:02.8725362Z set -ex 2024-08-20T22:13:02.8725659Z  2024-08-20T22:13:02.8726372Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:02.8727339Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:02.8728131Z # job could then download the pre-built image as usual 2024-08-20T22:13:02.8728883Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:02.8729594Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8730234Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8730795Z  2024-08-20T22:13:02.8731307Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:02.8731928Z  exit 0 2024-08-20T22:13:02.8732235Z else 2024-08-20T22:13:02.8732608Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8733061Z fi 2024-08-20T22:13:02.8733354Z  2024-08-20T22:13:02.8733831Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:02.8734673Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:02.8735452Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:02.8736163Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:02.8736886Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8737584Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8738136Z else 2024-08-20T22:13:02.8738578Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:02.8739243Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8740139Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8740880Z fi 2024-08-20T22:13:02.8752340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.8752842Z env: 2024-08-20T22:13:02.8753125Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.8753479Z REPO_NAME: pytorch 2024-08-20T22:13:02.8754602Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8755801Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:02.8756302Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.8756810Z ##[endgroup] 2024-08-20T22:13:02.8789687Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:02.8790256Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:02.8790636Z + echo skip=false 2024-08-20T22:13:02.8792683Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:02.8797065Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8798268Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:02.8825380Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8826195Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8827653Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8859648Z ##[group]Run set +e 2024-08-20T22:13:02.8860099Z set +e 2024-08-20T22:13:02.8860571Z set -x 2024-08-20T22:13:02.8860959Z  2024-08-20T22:13:02.8861457Z login() { 2024-08-20T22:13:02.8862279Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:02.8863086Z } 2024-08-20T22:13:02.8863478Z  2024-08-20T22:13:02.8863868Z retry () { 2024-08-20T22:13:02.8864364Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:02.8864925Z } 2024-08-20T22:13:02.8865348Z  2024-08-20T22:13:02.8865747Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:02.8866257Z  2024-08-20T22:13:02.8866875Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:02.8867700Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:02.8868259Z  exit 0 2024-08-20T22:13:02.8868803Z fi 2024-08-20T22:13:02.8869200Z  2024-08-20T22:13:02.8869840Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:02.8870727Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:02.8871574Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:02.8872382Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:02.8873092Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:02.8873579Z else 2024-08-20T22:13:02.8874194Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:02.8874995Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:02.8875586Z fi 2024-08-20T22:13:02.8875967Z  2024-08-20T22:13:02.8876406Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:02.8877026Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8877645Z  2024-08-20T22:13:02.8878390Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:02.8879238Z  exit 0 2024-08-20T22:13:02.8879668Z fi 2024-08-20T22:13:02.8880044Z  2024-08-20T22:13:02.8880575Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:02.8881694Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:02.8882542Z  exit 1 2024-08-20T22:13:02.8882957Z fi 2024-08-20T22:13:02.8883385Z  2024-08-20T22:13:02.8883973Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:02.8885002Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:02.8886011Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:02.8887069Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:02.8888321Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:02.8889212Z fi 2024-08-20T22:13:02.8889582Z  2024-08-20T22:13:02.8890086Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.8898584Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.8899272Z env: 2024-08-20T22:13:02.8899618Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.8900099Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:02.8900740Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:02.8902121Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8903432Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.8904193Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.8904906Z ##[endgroup] 2024-08-20T22:13:02.8934455Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.8935183Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.8936618Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:02.8938771Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.4282613Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:03.4283627Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:03.4284568Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:03.4285406Z 2024-08-20T22:13:03.4285589Z Login Succeeded 2024-08-20T22:13:03.4308218Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.6538121Z { 2024-08-20T22:13:03.6538548Z "schemaVersion": 2, 2024-08-20T22:13:03.6539881Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:03.6541072Z "config": { 2024-08-20T22:13:03.6541766Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:03.6542588Z "size": 48711, 2024-08-20T22:13:03.6544458Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:03.6545540Z }, 2024-08-20T22:13:03.6546012Z "layers": [ 2024-08-20T22:13:03.6546896Z { 2024-08-20T22:13:03.6547711Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6548872Z "size": 28580681, 2024-08-20T22:13:03.6549850Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:03.6550806Z }, 2024-08-20T22:13:03.6551249Z { 2024-08-20T22:13:03.6552855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6553913Z "size": 7943451, 2024-08-20T22:13:03.6555100Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:03.6556037Z }, 2024-08-20T22:13:03.6556556Z { 2024-08-20T22:13:03.6557202Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6558121Z "size": 55728572, 2024-08-20T22:13:03.6558960Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:03.6560227Z }, 2024-08-20T22:13:03.6560779Z { 2024-08-20T22:13:03.6561456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6562399Z "size": 186, 2024-08-20T22:13:03.6563245Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:03.6564155Z }, 2024-08-20T22:13:03.6564713Z { 2024-08-20T22:13:03.6565432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6566506Z "size": 6886, 2024-08-20T22:13:03.6567290Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:03.6568193Z }, 2024-08-20T22:13:03.6569045Z { 2024-08-20T22:13:03.6569745Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6570496Z "size": 1329001756, 2024-08-20T22:13:03.6571258Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:03.6571980Z }, 2024-08-20T22:13:03.6572285Z { 2024-08-20T22:13:03.6573261Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6573919Z "size": 62501, 2024-08-20T22:13:03.6574686Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:03.6575350Z }, 2024-08-20T22:13:03.6575691Z { 2024-08-20T22:13:03.6576298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6576944Z "size": 1684, 2024-08-20T22:13:03.6577562Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:03.6578516Z }, 2024-08-20T22:13:03.6578866Z { 2024-08-20T22:13:03.6579383Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6580086Z "size": 1523, 2024-08-20T22:13:03.6580734Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:03.6581522Z }, 2024-08-20T22:13:03.6581828Z { 2024-08-20T22:13:03.6582349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6583060Z "size": 2528295403, 2024-08-20T22:13:03.6583641Z + exit 0 2024-08-20T22:13:03.6584256Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:03.6585012Z }, 2024-08-20T22:13:03.6585358Z { 2024-08-20T22:13:03.6585912Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6586567Z "size": 86016, 2024-08-20T22:13:03.6587212Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:03.6587948Z }, 2024-08-20T22:13:03.6588299Z { 2024-08-20T22:13:03.6588970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6589612Z "size": 1822, 2024-08-20T22:13:03.6590393Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:03.6591157Z }, 2024-08-20T22:13:03.6591544Z { 2024-08-20T22:13:03.6592160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6593000Z "size": 246761563, 2024-08-20T22:13:03.6593804Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:03.6594515Z }, 2024-08-20T22:13:03.6594957Z { 2024-08-20T22:13:03.6595497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6596232Z "size": 544, 2024-08-20T22:13:03.6596810Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:03.6597507Z }, 2024-08-20T22:13:03.6598208Z { 2024-08-20T22:13:03.6598798Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6599791Z "size": 1258, 2024-08-20T22:13:03.6600488Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:03.6601250Z }, 2024-08-20T22:13:03.6601563Z { 2024-08-20T22:13:03.6602370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6603086Z "size": 484, 2024-08-20T22:13:03.6603727Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:03.6604412Z }, 2024-08-20T22:13:03.6604833Z { 2024-08-20T22:13:03.6605393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6606317Z "size": 91711766, 2024-08-20T22:13:03.6606954Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:03.6607740Z }, 2024-08-20T22:13:03.6608085Z { 2024-08-20T22:13:03.6608635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6609421Z "size": 3390, 2024-08-20T22:13:03.6610058Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:03.6610783Z }, 2024-08-20T22:13:03.6611139Z { 2024-08-20T22:13:03.6611669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6612326Z "size": 1909, 2024-08-20T22:13:03.6613015Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:03.6613696Z }, 2024-08-20T22:13:03.6614039Z { 2024-08-20T22:13:03.6614612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6615263Z "size": 701, 2024-08-20T22:13:03.6615853Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:03.6616615Z }, 2024-08-20T22:13:03.6617500Z { 2024-08-20T22:13:03.6618040Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6618739Z "size": 2864254879, 2024-08-20T22:13:03.6619417Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:03.6620133Z }, 2024-08-20T22:13:03.6620546Z { 2024-08-20T22:13:03.6621071Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6621710Z "size": 380, 2024-08-20T22:13:03.6622408Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:03.6623076Z }, 2024-08-20T22:13:03.6623454Z { 2024-08-20T22:13:03.6624043Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6624698Z "size": 12900, 2024-08-20T22:13:03.6625336Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:03.6626109Z }, 2024-08-20T22:13:03.6626447Z { 2024-08-20T22:13:03.6626988Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6627690Z "size": 803, 2024-08-20T22:13:03.6628330Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:03.6629253Z }, 2024-08-20T22:13:03.6629565Z { 2024-08-20T22:13:03.6630127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6630836Z "size": 106, 2024-08-20T22:13:03.6631459Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:03.6632140Z }, 2024-08-20T22:13:03.6632527Z { 2024-08-20T22:13:03.6633058Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6633722Z "size": 503, 2024-08-20T22:13:03.6634393Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:03.6635097Z }, 2024-08-20T22:13:03.6635465Z { 2024-08-20T22:13:03.6635999Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6636670Z "size": 121477267, 2024-08-20T22:13:03.6637325Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:03.6638075Z }, 2024-08-20T22:13:03.6638419Z { 2024-08-20T22:13:03.6638946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6639636Z "size": 109, 2024-08-20T22:13:03.6640286Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:03.6640983Z }, 2024-08-20T22:13:03.6641389Z { 2024-08-20T22:13:03.6641909Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6642511Z "size": 490, 2024-08-20T22:13:03.6643200Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:03.6643891Z }, 2024-08-20T22:13:03.6644290Z { 2024-08-20T22:13:03.6644799Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6645455Z "size": 297, 2024-08-20T22:13:03.6646122Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:03.6647003Z }, 2024-08-20T22:13:03.6647357Z { 2024-08-20T22:13:03.6647958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6648601Z "size": 103, 2024-08-20T22:13:03.6649235Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:03.6649978Z }, 2024-08-20T22:13:03.6650362Z { 2024-08-20T22:13:03.6650908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6651566Z "size": 1473, 2024-08-20T22:13:03.6652203Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:03.6652928Z }, 2024-08-20T22:13:03.6653347Z { 2024-08-20T22:13:03.6653837Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6654535Z "size": 594729516, 2024-08-20T22:13:03.6655383Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:03.6656092Z }, 2024-08-20T22:13:03.6656412Z { 2024-08-20T22:13:03.6656983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6657650Z "size": 164, 2024-08-20T22:13:03.6658245Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:03.6658988Z }, 2024-08-20T22:13:03.6659357Z { 2024-08-20T22:13:03.6659908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6660581Z "size": 564, 2024-08-20T22:13:03.6661259Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:03.6661956Z }, 2024-08-20T22:13:03.6662362Z { 2024-08-20T22:13:03.6662846Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6663485Z "size": 35874203, 2024-08-20T22:13:03.6664211Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:03.6664929Z }, 2024-08-20T22:13:03.6665235Z { 2024-08-20T22:13:03.6665830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6666472Z "size": 104, 2024-08-20T22:13:03.6667053Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:03.6667806Z }, 2024-08-20T22:13:03.6668189Z { 2024-08-20T22:13:03.6668935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6669555Z "size": 425, 2024-08-20T22:13:03.6670204Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:03.6671001Z }, 2024-08-20T22:13:03.6671342Z { 2024-08-20T22:13:03.6671830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6672563Z "size": 20262183, 2024-08-20T22:13:03.6673212Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:03.6673950Z }, 2024-08-20T22:13:03.6674328Z { 2024-08-20T22:13:03.6674898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6675562Z "size": 438, 2024-08-20T22:13:03.6676202Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:03.6676905Z }, 2024-08-20T22:13:03.6677262Z { 2024-08-20T22:13:03.6677831Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6678435Z "size": 701, 2024-08-20T22:13:03.6679082Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:03.6679835Z }, 2024-08-20T22:13:03.6680198Z { 2024-08-20T22:13:03.6680682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6681406Z "size": 141, 2024-08-20T22:13:03.6682061Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:03.6682746Z }, 2024-08-20T22:13:03.6683133Z { 2024-08-20T22:13:03.6683675Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6684415Z "size": 135, 2024-08-20T22:13:03.6685197Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:03.6685902Z }, 2024-08-20T22:13:03.6686239Z { 2024-08-20T22:13:03.6686842Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6687442Z "size": 4838068375, 2024-08-20T22:13:03.6688141Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:03.6688973Z }, 2024-08-20T22:13:03.6689278Z { 2024-08-20T22:13:03.6689830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6690515Z "size": 194, 2024-08-20T22:13:03.6691148Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:03.6691839Z }, 2024-08-20T22:13:03.6692233Z { 2024-08-20T22:13:03.6692882Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6693581Z "size": 564, 2024-08-20T22:13:03.6694185Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:03.6694907Z }, 2024-08-20T22:13:03.6695338Z { 2024-08-20T22:13:03.6695867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6696494Z "size": 43162921, 2024-08-20T22:13:03.6697189Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:03.6697891Z }, 2024-08-20T22:13:03.6698220Z { 2024-08-20T22:13:03.6698800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6699479Z "size": 106, 2024-08-20T22:13:03.6700111Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:03.6700846Z }, 2024-08-20T22:13:03.6701223Z { 2024-08-20T22:13:03.6701800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6702500Z "size": 1380, 2024-08-20T22:13:03.6703119Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:03.6703813Z }, 2024-08-20T22:13:03.6704216Z { 2024-08-20T22:13:03.6704730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6705435Z "size": 701, 2024-08-20T22:13:03.6706143Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:03.6706859Z }, 2024-08-20T22:13:03.6707177Z { 2024-08-20T22:13:03.6707785Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6708497Z "size": 139, 2024-08-20T22:13:03.6709319Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:03.6709985Z }, 2024-08-20T22:13:03.6710349Z { 2024-08-20T22:13:03.6710958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6711621Z "size": 118, 2024-08-20T22:13:03.6712220Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:03.6713010Z }, 2024-08-20T22:13:03.6713365Z { 2024-08-20T22:13:03.6713857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6714592Z "size": 4324672728, 2024-08-20T22:13:03.6715264Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:03.6716054Z }, 2024-08-20T22:13:03.6716427Z { 2024-08-20T22:13:03.6717299Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6717991Z "size": 174, 2024-08-20T22:13:03.6718672Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:03.6719332Z }, 2024-08-20T22:13:03.6719706Z { 2024-08-20T22:13:03.6720282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6720925Z "size": 907, 2024-08-20T22:13:03.6721554Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:03.6722470Z }, 2024-08-20T22:13:03.6722887Z { 2024-08-20T22:13:03.6723387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6724090Z "size": 701, 2024-08-20T22:13:03.6724748Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:03.6725469Z }, 2024-08-20T22:13:03.6725824Z { 2024-08-20T22:13:03.6726374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6727018Z "size": 134, 2024-08-20T22:13:03.6727734Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:03.6728416Z }, 2024-08-20T22:13:03.6728779Z { 2024-08-20T22:13:03.6729428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6730042Z "size": 32, 2024-08-20T22:13:03.6730802Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.6731586Z }, 2024-08-20T22:13:03.6731937Z { 2024-08-20T22:13:03.6732450Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6733154Z "size": 158, 2024-08-20T22:13:03.6733784Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:03.6734554Z }, 2024-08-20T22:13:03.6734855Z { 2024-08-20T22:13:03.6735379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6736094Z "size": 1841, 2024-08-20T22:13:03.6736783Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:03.6737458Z }, 2024-08-20T22:13:03.6737875Z { 2024-08-20T22:13:03.6738397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6739025Z "size": 7529776, 2024-08-20T22:13:03.6739722Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:03.6740436Z }, 2024-08-20T22:13:03.6740801Z { 2024-08-20T22:13:03.6741340Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6741983Z "size": 163, 2024-08-20T22:13:03.6742632Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:03.6743418Z }, 2024-08-20T22:13:03.6743725Z { 2024-08-20T22:13:03.6744277Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6744959Z "size": 7943, 2024-08-20T22:13:03.6745582Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:03.6746298Z }, 2024-08-20T22:13:03.6746681Z { 2024-08-20T22:13:03.6747224Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6747828Z "size": 8068, 2024-08-20T22:13:03.6748517Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:03.6749321Z }, 2024-08-20T22:13:03.6749677Z { 2024-08-20T22:13:03.6750285Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6750960Z "size": 302, 2024-08-20T22:13:03.6751590Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:03.6752362Z }, 2024-08-20T22:13:03.6752676Z { 2024-08-20T22:13:03.6753205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6753934Z "size": 7629638, 2024-08-20T22:13:03.6754540Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:03.6755333Z }, 2024-08-20T22:13:03.6755830Z { 2024-08-20T22:13:03.6756366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6757041Z "size": 108, 2024-08-20T22:13:03.6757778Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:03.6758494Z }, 2024-08-20T22:13:03.6759051Z { 2024-08-20T22:13:03.6759586Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6761094Z "size": 54145661, 2024-08-20T22:13:03.6761801Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:03.6762481Z }, 2024-08-20T22:13:03.6762919Z { 2024-08-20T22:13:03.6763409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6764113Z "size": 473, 2024-08-20T22:13:03.6764774Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:03.6765544Z }, 2024-08-20T22:13:03.6765898Z { 2024-08-20T22:13:03.6766456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6767101Z "size": 1374858766, 2024-08-20T22:13:03.6767802Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:03.6768481Z }, 2024-08-20T22:13:03.6768825Z { 2024-08-20T22:13:03.6769522Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6770164Z "size": 106, 2024-08-20T22:13:03.6770771Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:03.6771556Z }, 2024-08-20T22:13:03.6771974Z { 2024-08-20T22:13:03.6772472Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6773197Z "size": 559, 2024-08-20T22:13:03.6773835Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:03.6774566Z }, 2024-08-20T22:13:03.6774933Z { 2024-08-20T22:13:03.6775473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6776148Z "size": 46248488, 2024-08-20T22:13:03.6776876Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:03.6777549Z }, 2024-08-20T22:13:03.6777919Z { 2024-08-20T22:13:03.6778562Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6779172Z "size": 111, 2024-08-20T22:13:03.6779839Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:03.6780592Z }, 2024-08-20T22:13:03.6780958Z { 2024-08-20T22:13:03.6781443Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6782127Z "size": 32, 2024-08-20T22:13:03.6782785Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.6783542Z }, 2024-08-20T22:13:03.6783863Z { 2024-08-20T22:13:03.6784414Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6785131Z "size": 32, 2024-08-20T22:13:03.6785953Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.6786643Z }, 2024-08-20T22:13:03.6787065Z { 2024-08-20T22:13:03.6787624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6788240Z "size": 32, 2024-08-20T22:13:03.6789023Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.6789753Z }, 2024-08-20T22:13:03.6790101Z { 2024-08-20T22:13:03.6790643Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.6791317Z "size": 32, 2024-08-20T22:13:03.6791957Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.6792800Z } 2024-08-20T22:13:03.6793119Z ] 2024-08-20T22:13:03.6793472Z } 2024-08-20T22:13:03.7095415Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:03.7095926Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:03.7096483Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:03.7105644Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:03.7106153Z env: 2024-08-20T22:13:03.7106444Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:03.7107619Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.7108891Z ##[endgroup] 2024-08-20T22:13:03.7141167Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.7192257Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:03.7192839Z with: 2024-08-20T22:13:03.7193908Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.7195210Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.7195732Z env: 2024-08-20T22:13:03.7196016Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:03.7196370Z ##[endgroup] 2024-08-20T22:13:03.7213659Z ##[group]Run set -x 2024-08-20T22:13:03.7214178Z set -x 2024-08-20T22:13:03.7214485Z set +e 2024-08-20T22:13:03.7214784Z  2024-08-20T22:13:03.7215061Z login() { 2024-08-20T22:13:03.7215725Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:03.7216442Z } 2024-08-20T22:13:03.7216726Z  2024-08-20T22:13:03.7217304Z retry () { 2024-08-20T22:13:03.7217693Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:03.7218133Z } 2024-08-20T22:13:03.7218409Z  2024-08-20T22:13:03.7218725Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:03.7219137Z  2024-08-20T22:13:03.7219409Z set -e 2024-08-20T22:13:03.7219874Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:03.7220561Z # only pull docker image if it's not available locally 2024-08-20T22:13:03.7221307Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:03.7221997Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:03.7222430Z fi 2024-08-20T22:13:03.7230602Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:03.7231105Z env: 2024-08-20T22:13:03.7231399Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:03.7232549Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.7233856Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.7234383Z ##[endgroup] 2024-08-20T22:13:03.7261659Z + set +e 2024-08-20T22:13:03.7262331Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.7262979Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.7265619Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:03.7266730Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:04.2569170Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:04.2570029Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:04.2570991Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:04.2571605Z 2024-08-20T22:13:04.2571739Z Login Succeeded 2024-08-20T22:13:04.2591550Z + set -e 2024-08-20T22:13:04.2592869Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:04.2752893Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:04.2755011Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:04.5220772Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:04.5222020Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:04.5222517Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:04.5223333Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:04.5223843Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:04.5224358Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:04.5224867Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:04.5225381Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:04.5225888Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:04.5226367Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:04.5226882Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:04.5227371Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:04.5227896Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:04.5228409Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:04.5229163Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:04.5229643Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:04.5230101Z d527cbbb87e3: Waiting 2024-08-20T22:13:04.5230444Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:04.5230819Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:04.5231265Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:04.5231649Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:04.5232031Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:04.5232405Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:04.5232758Z b57676e46aee: Waiting 2024-08-20T22:13:04.5233098Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:04.5233475Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:04.5233834Z a8c1e85b5e14: Waiting 2024-08-20T22:13:04.5234154Z ec1e7978c1fe: Waiting 2024-08-20T22:13:04.5234470Z a41a8d1c11c8: Waiting 2024-08-20T22:13:04.5234794Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:04.5235121Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:04.5235491Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:04.5235852Z 0bf61114d37b: Waiting 2024-08-20T22:13:04.5236188Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:04.5236534Z 0c1227890755: Waiting 2024-08-20T22:13:04.5236861Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:04.5237218Z 2df849f72ba4: Waiting 2024-08-20T22:13:04.5237546Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:04.5237914Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:04.5238237Z 7ed32bc8e469: Waiting 2024-08-20T22:13:04.5238567Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:04.5238922Z ca426ffc9d2b: Waiting 2024-08-20T22:13:04.5239254Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:04.5239626Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:04.5239995Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:04.5240343Z d8d1234baab3: Waiting 2024-08-20T22:13:04.5240661Z 33b639226eee: Waiting 2024-08-20T22:13:04.5240976Z b119c4fc6c58: Waiting 2024-08-20T22:13:04.5241286Z 1fef4bd19792: Waiting 2024-08-20T22:13:04.5241619Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:04.5241982Z 34c47d63c5de: Waiting 2024-08-20T22:13:04.5242293Z 829ed7d91e38: Waiting 2024-08-20T22:13:04.5242617Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:04.5242993Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:04.5243352Z 75b2abb16c68: Waiting 2024-08-20T22:13:04.5243686Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:04.5244052Z 019620471c25: Pulling fs layer 2024-08-20T22:13:04.5244410Z d3547bd43146: Waiting 2024-08-20T22:13:04.5244741Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:04.5245108Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:04.5245477Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:04.5245848Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:04.5246218Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:04.5246586Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:04.5246956Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:04.5247311Z 0479fbac3361: Waiting 2024-08-20T22:13:04.5247635Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:04.5248005Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:04.5248375Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:04.5248724Z 019620471c25: Waiting 2024-08-20T22:13:04.5249045Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:04.5249441Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:04.5249826Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:04.5250196Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:04.5250543Z 560cd6122a38: Waiting 2024-08-20T22:13:04.5251001Z 6658e558ebe4: Waiting 2024-08-20T22:13:04.5251334Z f957c1624481: Pulling fs layer 2024-08-20T22:13:04.5251699Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:04.5252060Z f1d6874dee9c: Waiting 2024-08-20T22:13:04.5252393Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:04.5252752Z 7c47122f4e09: Waiting 2024-08-20T22:13:04.5253059Z 916c5e858016: Waiting 2024-08-20T22:13:04.5253385Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:04.5253742Z c42b3602a1fb: Waiting 2024-08-20T22:13:04.5254060Z 7754ee068b33: Waiting 2024-08-20T22:13:04.5254387Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:04.5254758Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:04.5255239Z cbb79a705353: Waiting 2024-08-20T22:13:04.5255564Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:04.5255920Z 768c31dca016: Waiting 2024-08-20T22:13:04.5256251Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:04.5256628Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:04.5256985Z afc429e87b59: Waiting 2024-08-20T22:13:04.5257310Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:04.5257645Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:04.5258021Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:04.5258381Z f957c1624481: Waiting 2024-08-20T22:13:04.5258695Z 0f085d4fa279: Waiting 2024-08-20T22:13:04.5259012Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:04.5259323Z 1811ff4d3556: Waiting 2024-08-20T22:13:04.5259633Z 4a0412373305: Waiting 2024-08-20T22:13:04.5259959Z ee1273153675: Pulling fs layer 2024-08-20T22:13:04.5260330Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:04.5260698Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:04.5261060Z a7a5392e2bf6: Waiting 2024-08-20T22:13:04.5261384Z e56eeff2db93: Waiting 2024-08-20T22:13:04.5261713Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:04.5262076Z b10de02255c0: Waiting 2024-08-20T22:13:04.5262406Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:04.5262763Z 37513e9b43d7: Waiting 2024-08-20T22:13:04.5263074Z 78ee77eddef2: Waiting 2024-08-20T22:13:04.5263407Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:04.5263786Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:04.5264141Z e22979f9e0ee: Waiting 2024-08-20T22:13:04.5264446Z ee1273153675: Waiting 2024-08-20T22:13:04.5264776Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:04.5265136Z eb09cbf3e416: Waiting 2024-08-20T22:13:04.5265461Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:04.5265820Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:04.5266136Z d79fd7d52698: Waiting 2024-08-20T22:13:04.5266448Z 829c0e19f4b7: Waiting 2024-08-20T22:13:04.5266758Z 7cbfce999d76: Waiting 2024-08-20T22:13:04.5267071Z 83808b2eff4d: Waiting 2024-08-20T22:13:04.5267383Z 5d115b29b5d5: Waiting 2024-08-20T22:13:04.5267700Z 9752d6fbbcab: Waiting 2024-08-20T22:13:04.5268015Z 02970347dff6: Waiting 2024-08-20T22:13:04.5268326Z 5a4eb19d7219: Waiting 2024-08-20T22:13:04.5268644Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:04.5269081Z 11fad972ef6c: Waiting 2024-08-20T22:13:04.5269394Z 4f4fb700ef54: Waiting 2024-08-20T22:13:04.5269710Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:04.5270033Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:04.5270343Z 33444a5a7e01: Waiting 2024-08-20T22:13:04.6456526Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:04.6457079Z 224fe954d725: Download complete 2024-08-20T22:13:04.7168338Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:04.7168952Z d527cbbb87e3: Download complete 2024-08-20T22:13:04.7877423Z b57676e46aee: Download complete 2024-08-20T22:13:04.8553169Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:04.8553604Z 7a2c55901189: Download complete 2024-08-20T22:13:04.9297964Z a41a8d1c11c8: Download complete 2024-08-20T22:13:04.9977404Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:04.9977902Z 0c1227890755: Download complete 2024-08-20T22:13:05.0681937Z d8d1234baab3: Verifying Checksum 2024-08-20T22:13:05.0682458Z d8d1234baab3: Download complete 2024-08-20T22:13:05.1344224Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:05.1344812Z 75722010b82e: Download complete 2024-08-20T22:13:05.2422614Z ec1e7978c1fe: Download complete 2024-08-20T22:13:05.3050476Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:06.0303585Z 7a2c55901189: Pull complete 2024-08-20T22:13:06.3526519Z 224fe954d725: Pull complete 2024-08-20T22:13:07.2694912Z 75722010b82e: Pull complete 2024-08-20T22:13:07.2896367Z d527cbbb87e3: Pull complete 2024-08-20T22:13:07.3105439Z b57676e46aee: Pull complete 2024-08-20T22:13:07.8165272Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:07.8165731Z ca426ffc9d2b: Download complete 2024-08-20T22:13:07.8827182Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:07.8827712Z 4373aa743caf: Download complete 2024-08-20T22:13:07.9503242Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:07.9503927Z 75b2abb16c68: Download complete 2024-08-20T22:13:08.0176997Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:08.9771304Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:08.9771773Z 0bf61114d37b: Download complete 2024-08-20T22:13:09.0733346Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:09.0733774Z 2df849f72ba4: Download complete 2024-08-20T22:13:09.1340602Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:09.2002927Z 1fef4bd19792: Download complete 2024-08-20T22:13:18.1249637Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:18.1250116Z a8c1e85b5e14: Download complete 2024-08-20T22:13:18.2092521Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:18.2093086Z b119c4fc6c58: Download complete 2024-08-20T22:13:18.2873515Z 33b639226eee: Download complete 2024-08-20T22:13:18.3732393Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:18.3733007Z d3547bd43146: Download complete 2024-08-20T22:13:18.4497311Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:18.4497901Z 5645520256fa: Download complete 2024-08-20T22:13:19.7885031Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:19.7885583Z 385d0421d67c: Download complete 2024-08-20T22:13:19.8744334Z 7cbfce999d76: Download complete 2024-08-20T22:13:19.9417312Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:19.9417790Z 6658e558ebe4: Download complete 2024-08-20T22:13:20.0319592Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:13:20.0320121Z 33444a5a7e01: Download complete 2024-08-20T22:13:20.1014770Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:20.1015233Z 7c47122f4e09: Download complete 2024-08-20T22:13:20.1805287Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:20.1805782Z 1811ff4d3556: Download complete 2024-08-20T22:13:26.1783341Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:26.1783931Z 7754ee068b33: Download complete 2024-08-20T22:13:26.2451870Z 0479fbac3361: Verifying Checksum 2024-08-20T22:13:26.2452494Z 0479fbac3361: Download complete 2024-08-20T22:13:26.3193543Z f1d6874dee9c: Download complete 2024-08-20T22:13:26.7601295Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:26.7601878Z afc429e87b59: Download complete 2024-08-20T22:13:26.8845462Z 019620471c25: Verifying Checksum 2024-08-20T22:13:26.8846001Z 019620471c25: Download complete 2024-08-20T22:13:26.9472106Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:26.9472617Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:27.1971580Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:27.1972180Z 829c0e19f4b7: Download complete 2024-08-20T22:13:27.2596505Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:27.2597060Z 560cd6122a38: Download complete 2024-08-20T22:13:27.3312773Z 5a4eb19d7219: Download complete 2024-08-20T22:13:27.3950505Z da6a838a66c7: Download complete 2024-08-20T22:13:30.4126580Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:30.4127195Z 7ed32bc8e469: Download complete 2024-08-20T22:13:30.4850929Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:13:30.4851542Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:30.5470342Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:13:30.5470854Z 0f085d4fa279: Download complete 2024-08-20T22:13:31.0376709Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:31.0377297Z 4a0412373305: Download complete 2024-08-20T22:13:31.0993613Z 302c8c177944: Download complete 2024-08-20T22:13:31.1719151Z 02970347dff6: Download complete 2024-08-20T22:13:31.2466914Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:31.2467775Z 916c5e858016: Download complete 2024-08-20T22:13:31.3172228Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:31.3172789Z c42b3602a1fb: Download complete 2024-08-20T22:13:31.3510741Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:31.5359952Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:31.7039779Z 0c1227890755: Pull complete 2024-08-20T22:13:31.8445383Z d8d1234baab3: Pull complete 2024-08-20T22:13:37.8957507Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:37.8957975Z 6efb96fc87ea: Download complete 2024-08-20T22:13:38.0171223Z f957c1624481: Verifying Checksum 2024-08-20T22:13:38.0171800Z f957c1624481: Download complete 2024-08-20T22:13:38.0807995Z 768c31dca016: Verifying Checksum 2024-08-20T22:13:38.0808496Z 768c31dca016: Download complete 2024-08-20T22:13:38.1488438Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:38.1563268Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:13:38.1563824Z 4f4fb700ef54: Download complete 2024-08-20T22:13:38.2261672Z 78ee77eddef2: Download complete 2024-08-20T22:13:38.2867665Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:13:38.2868236Z 37513e9b43d7: Download complete 2024-08-20T22:13:38.4243112Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:38.4243773Z a7a5392e2bf6: Download complete 2024-08-20T22:13:38.5158792Z b10de02255c0: Download complete 2024-08-20T22:13:38.5824466Z e56eeff2db93: Verifying Checksum 2024-08-20T22:13:38.5824924Z e56eeff2db93: Download complete 2024-08-20T22:13:38.6509638Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:13:38.6510086Z 70ac3e38d0fd: Download complete 2024-08-20T22:13:38.7213513Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:13:38.7214100Z eb09cbf3e416: Download complete 2024-08-20T22:13:38.8579179Z ee1273153675: Verifying Checksum 2024-08-20T22:13:38.8579725Z ee1273153675: Download complete 2024-08-20T22:13:38.9234323Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:13:38.9234940Z e22979f9e0ee: Download complete 2024-08-20T22:13:39.5096541Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:39.5097157Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:39.6040817Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:13:39.6041401Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:05.3595132Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:05.3595592Z 5d115b29b5d5: Download complete 2024-08-20T22:14:05.4426774Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:05.4427209Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:05.5128418Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:05.5128854Z d79fd7d52698: Download complete 2024-08-20T22:14:07.0437632Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:07.0438195Z 9752d6fbbcab: Download complete 2024-08-20T22:14:07.1171882Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:14:07.1172469Z 83808b2eff4d: Download complete 2024-08-20T22:14:44.9632945Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:14:44.9633508Z 11fad972ef6c: Download complete 2024-08-20T22:14:46.4455651Z cbb79a705353: Verifying Checksum 2024-08-20T22:14:46.4456228Z cbb79a705353: Download complete 2024-08-20T22:15:01.0727643Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:01.3095418Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:01.5427371Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:10.1578845Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:10.3869446Z 4373aa743caf: Pull complete 2024-08-20T22:15:10.6178587Z 75b2abb16c68: Pull complete 2024-08-20T22:15:10.8394938Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:13.4007437Z 0bf61114d37b: Pull complete 2024-08-20T22:15:13.5466186Z 2df849f72ba4: Pull complete 2024-08-20T22:15:13.6602381Z 34c47d63c5de: Pull complete 2024-08-20T22:15:13.7369914Z 1fef4bd19792: Pull complete 2024-08-20T22:16:12.1665203Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:12.3370789Z b119c4fc6c58: Pull complete 2024-08-20T22:16:12.5125453Z 33b639226eee: Pull complete 2024-08-20T22:16:12.6491269Z d3547bd43146: Pull complete 2024-08-20T22:16:12.7634285Z 5645520256fa: Pull complete 2024-08-20T22:16:12.8835703Z 829ed7d91e38: Pull complete 2024-08-20T22:16:16.3438722Z 385d0421d67c: Pull complete 2024-08-20T22:16:16.5197440Z 7cbfce999d76: Pull complete 2024-08-20T22:16:16.6346560Z 6658e558ebe4: Pull complete 2024-08-20T22:16:16.8506171Z 33444a5a7e01: Pull complete 2024-08-20T22:16:17.0700524Z 7c47122f4e09: Pull complete 2024-08-20T22:16:17.2487101Z 1811ff4d3556: Pull complete 2024-08-20T22:16:27.8894977Z 7754ee068b33: Pull complete 2024-08-20T22:16:28.1047108Z 0479fbac3361: Pull complete 2024-08-20T22:16:28.3304051Z f1d6874dee9c: Pull complete 2024-08-20T22:16:29.3221023Z afc429e87b59: Pull complete 2024-08-20T22:16:29.5572849Z 019620471c25: Pull complete 2024-08-20T22:16:29.7422710Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:30.2258852Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:30.4315484Z 560cd6122a38: Pull complete 2024-08-20T22:16:30.8576363Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:31.0530877Z da6a838a66c7: Pull complete 2024-08-20T22:17:10.0137666Z cbb79a705353: Pull complete 2024-08-20T22:17:10.2349765Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:10.4739964Z 0f085d4fa279: Pull complete 2024-08-20T22:17:12.9101781Z 4a0412373305: Pull complete 2024-08-20T22:17:12.9225349Z 302c8c177944: Pull complete 2024-08-20T22:17:12.9360665Z 02970347dff6: Pull complete 2024-08-20T22:17:12.9637746Z 916c5e858016: Pull complete 2024-08-20T22:17:12.9761544Z c42b3602a1fb: Pull complete 2024-08-20T22:18:34.8790535Z 11fad972ef6c: Pull complete 2024-08-20T22:18:35.0927724Z f957c1624481: Pull complete 2024-08-20T22:18:35.3343619Z 768c31dca016: Pull complete 2024-08-20T22:18:35.7366520Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:35.9658249Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:36.1910370Z 78ee77eddef2: Pull complete 2024-08-20T22:18:36.3757893Z 37513e9b43d7: Pull complete 2024-08-20T22:18:36.7519068Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:36.9759110Z b10de02255c0: Pull complete 2024-08-20T22:18:37.2007590Z e56eeff2db93: Pull complete 2024-08-20T22:18:37.4257010Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:37.6481726Z eb09cbf3e416: Pull complete 2024-08-20T22:18:39.0645842Z ee1273153675: Pull complete 2024-08-20T22:18:39.2853298Z e22979f9e0ee: Pull complete 2024-08-20T22:18:41.4764243Z 6c1f0aabce7c: Pull complete 2024-08-20T22:18:41.7055528Z 7ebe49c3a23e: Pull complete 2024-08-20T22:18:55.1559102Z 5d115b29b5d5: Pull complete 2024-08-20T22:18:55.3659863Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:18:55.5638866Z d79fd7d52698: Pull complete 2024-08-20T22:18:56.4672328Z 9752d6fbbcab: Pull complete 2024-08-20T22:18:56.7022370Z 83808b2eff4d: Pull complete 2024-08-20T22:18:57.7204382Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:18:57.7685787Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:18:57.7932161Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:18:57.8006292Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:18:57.8007228Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:18:57.8017656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:57.8018169Z env: 2024-08-20T22:18:57.8018458Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:57.8018818Z ##[endgroup] 2024-08-20T22:18:57.8224967Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:18:57.8225526Z with: 2024-08-20T22:18:57.8225807Z driver-version: 550.54.15 2024-08-20T22:18:57.8226138Z env: 2024-08-20T22:18:57.8226409Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:57.8226768Z ##[endgroup] 2024-08-20T22:18:57.8303810Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:18:57.8304368Z with: 2024-08-20T22:18:57.8304648Z timeout_minutes: 10 2024-08-20T22:18:57.8304974Z max_attempts: 3 2024-08-20T22:18:57.8337268Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:18:57.8369885Z retry_wait_seconds: 10 2024-08-20T22:18:57.8370259Z polling_interval_seconds: 1 2024-08-20T22:18:57.8370634Z warning_on_retry: true 2024-08-20T22:18:57.8370993Z continue_on_error: false 2024-08-20T22:18:57.8371334Z env: 2024-08-20T22:18:57.8371620Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:57.8371973Z DRIVER_VERSION: 550.54.15 2024-08-20T22:18:57.8372327Z ##[endgroup] 2024-08-20T22:18:57.9180431Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:18:57.9181271Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:18:57.9182997Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:18:58.3835934Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:18:58.3836576Z No packages marked for removal. 2024-08-20T22:18:58.3898628Z Dependencies resolved. 2024-08-20T22:18:58.3907907Z Nothing to do. 2024-08-20T22:18:58.3908311Z Complete! 2024-08-20T22:18:58.4784659Z + install_nvidia_driver_common 2024-08-20T22:18:58.4788412Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:18:58.4788985Z + lspci 2024-08-20T22:18:58.4789293Z Before installing NVIDIA driver 2024-08-20T22:18:58.5912604Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:18:58.5913506Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:18:58.5914460Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:18:58.5915257Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:18:58.5916063Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:18:58.5916855Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:18:58.5918029Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:18:58.5919019Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:18:58.5919770Z + lsmod 2024-08-20T22:18:58.5958770Z Module Size Used by 2024-08-20T22:18:58.5959350Z xt_conntrack 16384 1 2024-08-20T22:18:58.5959866Z nft_chain_nat 16384 3 2024-08-20T22:18:58.5960387Z xt_MASQUERADE 20480 1 2024-08-20T22:18:58.5960902Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:18:58.5961505Z nf_conntrack_netlink 57344 0 2024-08-20T22:18:58.5962275Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:18:58.5963020Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:18:58.5963500Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:18:58.5963938Z xfrm_user 57344 1 2024-08-20T22:18:58.5964338Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:18:58.5964824Z xt_addrtype 16384 2 2024-08-20T22:18:58.5965294Z nft_compat 20480 4 2024-08-20T22:18:58.5965758Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:18:58.5966373Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:18:58.5966916Z br_netfilter 36864 0 2024-08-20T22:18:58.5967329Z bridge 307200 1 br_netfilter 2024-08-20T22:18:58.5967791Z stp 16384 1 bridge 2024-08-20T22:18:58.5968407Z llc 16384 2 bridge,stp 2024-08-20T22:18:58.5968837Z overlay 167936 0 2024-08-20T22:18:58.5969200Z tls 114688 0 2024-08-20T22:18:58.5969570Z nls_ascii 16384 1 2024-08-20T22:18:58.5969940Z nls_cp437 20480 1 2024-08-20T22:18:58.5970306Z vfat 24576 1 2024-08-20T22:18:58.5970670Z fat 86016 1 vfat 2024-08-20T22:18:58.5971068Z ena 167936 0 2024-08-20T22:18:58.5971441Z ghash_clmulni_intel 16384 0 2024-08-20T22:18:58.5971829Z ptp 36864 1 ena 2024-08-20T22:18:58.5972225Z pps_core 24576 1 ptp 2024-08-20T22:18:58.5972622Z aesni_intel 393216 0 2024-08-20T22:18:58.5973001Z i8042 45056 0 2024-08-20T22:18:58.5973373Z serio 28672 3 i8042 2024-08-20T22:18:58.5973799Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:18:58.5974323Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:18:58.5974834Z button 24576 0 2024-08-20T22:18:58.5975202Z sunrpc 692224 1 2024-08-20T22:18:58.5975597Z sch_fq_codel 20480 17 2024-08-20T22:18:58.5975976Z dm_mod 188416 0 2024-08-20T22:18:58.5976346Z fuse 163840 1 2024-08-20T22:18:58.5976706Z loop 36864 0 2024-08-20T22:18:58.5977078Z configfs 57344 1 2024-08-20T22:18:58.5977459Z dax 45056 1 dm_mod 2024-08-20T22:18:58.5977864Z dmi_sysfs 20480 0 2024-08-20T22:18:58.5978232Z crc32_pclmul 16384 0 2024-08-20T22:18:58.5978609Z crc32c_intel 24576 0 2024-08-20T22:18:58.5979025Z efivarfs 24576 1 2024-08-20T22:18:58.5979514Z + modinfo nvidia 2024-08-20T22:18:58.5980480Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:18:58.5981422Z alias: char-major-195-* 2024-08-20T22:18:58.5981937Z version: 550.54.15 2024-08-20T22:18:58.5982290Z supported: external 2024-08-20T22:18:58.5982632Z license: NVIDIA 2024-08-20T22:18:58.5983008Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:18:58.5983487Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:18:58.5983938Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:18:58.5984445Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:18:58.5984963Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:18:58.5985447Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:18:58.5985945Z depends: i2c-core,drm 2024-08-20T22:18:58.5986305Z retpoline: Y 2024-08-20T22:18:58.5986677Z name: nvidia 2024-08-20T22:18:58.5987686Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:18:58.5988741Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:18:58.5989607Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:18:58.5990256Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:18:58.5990711Z parm: NVreg_RmLogonRC:int 2024-08-20T22:18:58.5991150Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:18:58.5991617Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:18:58.5992062Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:18:58.5992503Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:18:58.5993043Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:18:58.5993616Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:18:58.5994105Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:18:58.5994541Z parm: NVreg_EnableMSI:int 2024-08-20T22:18:58.5995026Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:18:58.5995492Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:18:58.5996067Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:18:58.5996662Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:18:58.5997349Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:18:58.5997967Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:18:58.5998576Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:18:58.5999199Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:18:58.5999797Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:18:58.6000300Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:18:58.6000841Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:18:58.6001401Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:18:58.6001902Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:18:58.6002385Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:18:58.6002886Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:18:58.6003364Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:18:58.6003824Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:18:58.6004346Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:18:58.6004880Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:18:58.6005362Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:18:58.6005861Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:18:58.6006373Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:18:58.6006871Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:18:58.6007368Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:18:58.6007850Z parm: NVreg_RmMsg:charp 2024-08-20T22:18:58.6008268Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:18:58.6008742Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:18:58.6009216Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:18:58.6009685Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:18:58.6010172Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:18:58.6010654Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:18:58.6011112Z parm: rm_firmware_active:charp 2024-08-20T22:18:58.6011529Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:18:58.6011953Z ++ command -v nvidia-smi 2024-08-20T22:18:58.6012359Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:18:58.6012740Z + set +e 2024-08-20T22:18:58.6013262Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:00.8564702Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:00.8565207Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:00.8565848Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:00.8566231Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:00.8566601Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:00.8567693Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:00.8568409Z + set -e 2024-08-20T22:19:00.8568724Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:00.8569298Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:00.8569996Z + post_install_nvidia_driver_common 2024-08-20T22:19:00.8570409Z + sudo modprobe nvidia 2024-08-20T22:19:00.9811967Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:00.9812527Z + lspci 2024-08-20T22:19:00.9812854Z After installing NVIDIA driver 2024-08-20T22:19:00.9922236Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:00.9923198Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:00.9924410Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:00.9925479Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:00.9926529Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:00.9927546Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:00.9928377Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:00.9929190Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:00.9930045Z + lsmod 2024-08-20T22:19:00.9954364Z Module Size Used by 2024-08-20T22:19:00.9954845Z nvidia_uvm 4706304 0 2024-08-20T22:19:00.9955530Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:19:00.9956082Z drm 602112 1 nvidia 2024-08-20T22:19:00.9956684Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:00.9957253Z backlight 24576 1 drm 2024-08-20T22:19:00.9957675Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:00.9958092Z xt_conntrack 16384 1 2024-08-20T22:19:00.9958466Z nft_chain_nat 16384 3 2024-08-20T22:19:00.9958833Z xt_MASQUERADE 20480 1 2024-08-20T22:19:00.9959295Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:00.9959781Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:00.9960353Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:00.9960990Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:00.9961450Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:00.9961870Z xfrm_user 57344 1 2024-08-20T22:19:00.9962266Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:00.9962676Z xt_addrtype 16384 2 2024-08-20T22:19:00.9963050Z nft_compat 20480 4 2024-08-20T22:19:00.9963502Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:00.9964102Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:00.9964642Z br_netfilter 36864 0 2024-08-20T22:19:00.9965047Z bridge 307200 1 br_netfilter 2024-08-20T22:19:00.9965539Z stp 16384 1 bridge 2024-08-20T22:19:00.9965944Z llc 16384 2 bridge,stp 2024-08-20T22:19:00.9966357Z overlay 167936 0 2024-08-20T22:19:00.9966716Z tls 114688 0 2024-08-20T22:19:00.9967078Z nls_ascii 16384 1 2024-08-20T22:19:00.9967437Z nls_cp437 20480 1 2024-08-20T22:19:00.9967800Z vfat 24576 1 2024-08-20T22:19:00.9968166Z fat 86016 1 vfat 2024-08-20T22:19:00.9968552Z ena 167936 0 2024-08-20T22:19:00.9968910Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:00.9969292Z ptp 36864 1 ena 2024-08-20T22:19:00.9969692Z pps_core 24576 1 ptp 2024-08-20T22:19:00.9970073Z aesni_intel 393216 0 2024-08-20T22:19:00.9970434Z i8042 45056 0 2024-08-20T22:19:00.9970805Z serio 28672 3 i8042 2024-08-20T22:19:00.9971224Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:00.9971963Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:00.9972464Z button 24576 0 2024-08-20T22:19:00.9972829Z sunrpc 692224 1 2024-08-20T22:19:00.9973202Z sch_fq_codel 20480 17 2024-08-20T22:19:00.9973645Z dm_mod 188416 0 2024-08-20T22:19:00.9974115Z fuse 163840 1 2024-08-20T22:19:00.9975790Z loop 36864 0 2024-08-20T22:19:00.9976157Z configfs 57344 1 2024-08-20T22:19:00.9976539Z dax 45056 1 dm_mod 2024-08-20T22:19:00.9976931Z dmi_sysfs 20480 0 2024-08-20T22:19:00.9977293Z crc32_pclmul 16384 0 2024-08-20T22:19:00.9977645Z crc32c_intel 24576 0 2024-08-20T22:19:00.9978008Z efivarfs 24576 1 2024-08-20T22:19:00.9978382Z + modinfo nvidia 2024-08-20T22:19:00.9979150Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:00.9979851Z alias: char-major-195-* 2024-08-20T22:19:00.9980222Z version: 550.54.15 2024-08-20T22:19:00.9980568Z supported: external 2024-08-20T22:19:00.9980908Z license: NVIDIA 2024-08-20T22:19:00.9981270Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:00.9981895Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:00.9982346Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:00.9982804Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:00.9983290Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:00.9983777Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:00.9984267Z depends: i2c-core,drm 2024-08-20T22:19:00.9984618Z retpoline: Y 2024-08-20T22:19:00.9984912Z name: nvidia 2024-08-20T22:19:00.9985565Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:00.9986257Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:00.9986899Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:00.9987474Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:00.9987911Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:00.9988336Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:00.9988945Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:00.9989369Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:00.9989803Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:00.9990319Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:00.9990860Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:00.9991335Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:00.9991759Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:00.9992170Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:00.9992618Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:00.9993141Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:00.9993722Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:00.9994274Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:00.9994868Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:00.9995510Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:00.9996119Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:00.9996705Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:00.9997178Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:00.9997705Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:00.9998243Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:00.9998734Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:00.9999186Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:00.9999660Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:01.0000125Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:01.0000670Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:01.0001166Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:01.0001685Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:01.0002153Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:01.0002632Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:01.0003127Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:01.0003605Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:01.0004096Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:01.0004562Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:01.0004965Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:01.0005426Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:01.0005889Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:01.0006330Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:01.0006806Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:01.0007275Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:01.0007719Z parm: rm_firmware_active:charp 2024-08-20T22:19:01.0008109Z + set +e 2024-08-20T22:19:01.0008415Z + nvidia-smi 2024-08-20T22:19:02.5785142Z Tue Aug 20 22:19:02 2024 2024-08-20T22:19:02.5787104Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:02.5788920Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:02.5790518Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:02.5792142Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:02.5793958Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:02.5795353Z | | | MIG M. | 2024-08-20T22:19:02.5795981Z |=========================================+========================+======================| 2024-08-20T22:19:02.5883951Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:02.5884763Z | 0% 29C P0 59W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:02.5885416Z | | | N/A | 2024-08-20T22:19:02.5886164Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:02.5886756Z 2024-08-20T22:19:02.5887553Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:02.5888314Z | Processes: | 2024-08-20T22:19:02.5889021Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:02.5889724Z | ID ID Usage | 2024-08-20T22:19:02.5890292Z |=========================================================================================| 2024-08-20T22:19:02.5890961Z | No running processes found | 2024-08-20T22:19:02.5891732Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:03.2082715Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:04.7726880Z NVIDIA A10G 2024-08-20T22:19:05.2360336Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:05.2361054Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:05.2361582Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:05.2362093Z + set -e 2024-08-20T22:19:05.2362490Z INFO: Ignoring allowed status 0 2024-08-20T22:19:05.2369889Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:05.2374098Z + sudo yum install -y yum-utils 2024-08-20T22:19:05.6606949Z Last metadata expiration check: 0:09:37 ago on Tue Aug 20 22:09:28 2024. 2024-08-20T22:19:05.6828046Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:05.7129175Z Dependencies resolved. 2024-08-20T22:19:05.7255183Z Nothing to do. 2024-08-20T22:19:05.7255827Z Complete! 2024-08-20T22:19:05.8150629Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:05.8151717Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:05.8153017Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:06.1292723Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:06.2041632Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:06.6691971Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:19:06.6911229Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:06.7206744Z Dependencies resolved. 2024-08-20T22:19:06.7331075Z Nothing to do. 2024-08-20T22:19:06.7331416Z Complete! 2024-08-20T22:19:06.8290405Z + sudo systemctl restart docker 2024-08-20T22:20:00.9375553Z Tue Aug 20 22:20:00 2024 2024-08-20T22:20:00.9376465Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:00.9377365Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:00.9378428Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:00.9379401Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:00.9380298Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:00.9381017Z | | | MIG M. | 2024-08-20T22:20:00.9381558Z |=========================================+========================+======================| 2024-08-20T22:20:00.9497706Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:00.9498562Z | 0% 29C P0 59W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:20:00.9499284Z | | | N/A | 2024-08-20T22:20:00.9500127Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:00.9500801Z 2024-08-20T22:20:00.9501555Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:00.9502266Z | Processes: | 2024-08-20T22:20:00.9503063Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:00.9503820Z | ID ID Usage | 2024-08-20T22:20:00.9504450Z |=========================================================================================| 2024-08-20T22:20:00.9505163Z | No running processes found | 2024-08-20T22:20:00.9506021Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:01.9559246Z Command completed after 1 attempt(s). 2024-08-20T22:20:01.9640835Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:01.9641610Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:01.9642302Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:01.9642955Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:01.9657239Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:01.9657752Z env: 2024-08-20T22:20:01.9658039Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:01.9658520Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:01.9659013Z ##[endgroup] 2024-08-20T22:20:02.2675482Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:02.6933264Z Collecting psutil==5.9.1 2024-08-20T22:20:02.7343998Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:02.7917994Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:02.7970620Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:02.8899453Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:03.0471880Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:03.2476156Z Prepare all required actions 2024-08-20T22:20:03.2476890Z Getting action download info 2024-08-20T22:20:03.3645377Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:03.5627456Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:03.6781584Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:03.6782066Z with: 2024-08-20T22:20:03.6782403Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:03.6782860Z s3-bucket: gha-artifacts 2024-08-20T22:20:03.6783199Z env: 2024-08-20T22:20:03.6783481Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:03.6783928Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:03.6784421Z ##[endgroup] 2024-08-20T22:20:03.6818852Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:03.6819320Z with: 2024-08-20T22:20:03.6819725Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:03.6820182Z s3-bucket: gha-artifacts 2024-08-20T22:20:03.6820527Z region: us-east-1 2024-08-20T22:20:03.6820825Z env: 2024-08-20T22:20:03.6821098Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:03.6821543Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:03.6822043Z ##[endgroup] 2024-08-20T22:20:04.1505354Z (node:52581) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:04.1506712Z 2024-08-20T22:20:04.1507229Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:04.1508693Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:04.1509613Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:04.2113842Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:04.2115985Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0751045Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0757955Z Artifact download has finished successfully 2024-08-20T22:20:25.1111215Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:25.1111671Z unzip -o artifacts.zip 2024-08-20T22:20:25.1120962Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:25.1121470Z env: 2024-08-20T22:20:25.1121749Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:25.1122208Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:25.1122695Z ##[endgroup] 2024-08-20T22:20:25.1195579Z Archive: artifacts.zip 2024-08-20T22:20:25.1197193Z creating: dist/ 2024-08-20T22:20:27.2471860Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:27.2472741Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:27.2473574Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:27.2474616Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:27.2475782Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.2482179Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.2483497Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.2484791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.2486253Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.2487584Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.2489170Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.2491019Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.2492488Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.2493873Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.2495788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.2497436Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.2499495Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.2501476Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.2503839Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.2505685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.2507134Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.2508624Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.2549475Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.2589776Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.2591901Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.2638432Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.2640379Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.2642359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.2644386Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.2646331Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.2648304Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.2650218Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.2652142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.2654039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.2655846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.2657614Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.2659447Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.2661211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.2662881Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.2664658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.2720482Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.2781043Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.2782656Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.2784291Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.2785515Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.2786779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.2788057Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:27.2789596Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2791243Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2792789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:27.2794223Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:27.2795770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2797340Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:27.2798842Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2800405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:27.2801868Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:27.2818325Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:27.2962947Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:27.2964356Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:27.2965874Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2967548Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2969186Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:27.2970714Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:27.2972278Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2973857Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:27.2975548Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2977143Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:27.2978776Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:27.2997021Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:27.3081287Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:27.3083220Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3084724Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3086301Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3087582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3088855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3090339Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:27.3091778Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:27.3093310Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:27.3094671Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:27.3215597Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:27.3279645Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:27.3280687Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:27.3281719Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:27.3282899Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3289549Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3290839Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3292103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3293533Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3294851Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3296397Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3297874Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3299253Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3300669Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3302467Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3304645Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3307119Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3308843Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3310618Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3312445Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3313942Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3315398Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3355977Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3395163Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3396614Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3442126Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3443535Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3445266Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3446739Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3448451Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3449818Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3451197Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3452555Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3453917Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3455194Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3456414Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3457622Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3458834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3460022Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3461219Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3516678Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.3578153Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.3580478Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.3582433Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.3584148Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.3585167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.3586105Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:27.3587159Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:27.3588338Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:27.3589549Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:27.3590616Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:27.3591720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:27.3592822Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:27.3593922Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:27.3595017Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:27.3596120Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:27.3609178Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:27.3673731Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:27.3674918Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3675989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3677086Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3677989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3678887Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3679779Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:27.3681935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:27.3683033Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:27.3683890Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:27.3733639Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:27.3734405Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:27.3735177Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:27.3736067Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3742498Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3743501Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3744498Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3745549Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3746580Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3747738Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3749170Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3750258Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3751319Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3752761Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3754140Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3756044Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3757242Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3758853Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3760221Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3761316Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3762374Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3802108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3841871Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3843559Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3888593Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3890225Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3891721Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3893257Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3894776Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3896228Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3897700Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3899144Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3900547Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3901877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3903177Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3904460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3905749Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3907000Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3908266Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3963880Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.4025163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.4026457Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.4027508Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.4028424Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.4029473Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.4030490Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:27.4031615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.4032873Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.4034070Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:27.4035211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:27.4036398Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.4037741Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:27.4038935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.4040108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:27.4041423Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:27.4042671Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:27.4166480Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:27.4168816Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:27.4171167Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.4173721Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.4175032Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:27.4176236Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:27.4177469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.4178690Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:27.4179926Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.4181155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:27.4182372Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:27.4194869Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:27.4250531Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:27.4251819Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.4252949Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.4253987Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.4254944Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.4255899Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.4256854Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:27.4258470Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:27.4259643Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:27.4260628Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:27.4364299Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:27.4407324Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:27.4407962Z creating: build/lib/ 2024-08-20T22:20:27.4497548Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:27.4952732Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:27.4962464Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:27.4970745Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:27.4978970Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:27.4979608Z inflating: build/lib/libclog.a 2024-08-20T22:20:27.4998236Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:27.5000738Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:27.5066284Z inflating: build/lib/libgtest.a 2024-08-20T22:20:27.5141181Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:27.5204637Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:27.5212047Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:27.5241426Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:27.5369332Z inflating: build/lib/libgloo.a 2024-08-20T22:20:27.5389293Z inflating: build/lib/libfmt.a 2024-08-20T22:20:27.5484206Z inflating: build/lib/libc10.so 2024-08-20T22:20:27.5486080Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:27.5487175Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:27.5507284Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:27.6009312Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:27.6027640Z inflating: build/lib/libgmock.a 2024-08-20T22:20:27.6028169Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:27.6029024Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:27.6418485Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:28.6579850Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:28.7152822Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:28.7209350Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:28.7209936Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:28.8467289Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:28.8723438Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:28.9229435Z inflating: build/lib/libkineto.a 2024-08-20T22:20:28.9414047Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:28.9456718Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:29.0147941Z inflating: build/lib/libonnx.a 2024-08-20T22:20:31.4595728Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:31.4601718Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:31.4605503Z inflating: build/lib/libshm.so 2024-08-20T22:20:33.5234236Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:33.5235835Z inflating: build/lib/libtorch.so 2024-08-20T22:20:33.5238515Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:34.3415111Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:34.5410778Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:34.5482905Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:34.5504086Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:34.5530764Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:34.5556502Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:34.5591419Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:34.5592030Z creating: build/bin/ 2024-08-20T22:20:34.5642621Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:34.5694889Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:34.5746014Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:34.5806069Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:34.5860335Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:34.5909156Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:34.5960913Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:34.6016017Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:34.6072040Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:34.6129166Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:34.6200516Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:34.6254761Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:34.6303515Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:34.6354591Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:34.6405131Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:34.6462030Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:34.6517596Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:34.6568243Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:34.6624247Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:34.6676229Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:34.6727707Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:34.6777239Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:34.6829356Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:34.6885110Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:34.6936561Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:34.6993151Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:34.7050087Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:34.7102478Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:34.7153849Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:34.7203995Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:34.7370030Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:34.7420745Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:34.7474970Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:34.7532890Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:34.7595774Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:34.7671406Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:34.7823189Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:34.7877413Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:34.7929699Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:34.7982028Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:34.8033071Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:34.8092148Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:34.8141599Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:34.8197990Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:34.8646186Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:34.9093857Z inflating: build/bin/protoc 2024-08-20T22:20:34.9146842Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:34.9200346Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:34.9254305Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:34.9306165Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:34.9359531Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:34.9412810Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:34.9462446Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:34.9516246Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:34.9847976Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:35.0193933Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:35.0554989Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:35.0607726Z inflating: build/bin/BackoffTest 2024-08-20T22:20:35.0661739Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:35.0717636Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:35.0771461Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:35.0785458Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:35.0840005Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:35.0844607Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:35.0847994Z inflating: build/bin/example_allreduce 2024-08-20T22:20:35.0904339Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:35.0973678Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:35.0976823Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:35.1044708Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:35.1053988Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:35.1401604Z inflating: build/bin/test_lazy 2024-08-20T22:20:35.2586789Z inflating: build/bin/test_api 2024-08-20T22:20:35.2660868Z inflating: build/bin/Dict_test 2024-08-20T22:20:35.2713166Z inflating: build/bin/Dimname_test 2024-08-20T22:20:35.2778157Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:35.2836061Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:35.2896033Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:35.2955541Z inflating: build/bin/atest 2024-08-20T22:20:35.3018699Z inflating: build/bin/basic 2024-08-20T22:20:35.3074189Z inflating: build/bin/broadcast_test 2024-08-20T22:20:35.3125528Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:35.3184741Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:35.3238158Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:35.3331993Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:35.3382457Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:35.3433513Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:35.3492675Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:35.3548030Z inflating: build/bin/half_test 2024-08-20T22:20:35.3644968Z inflating: build/bin/ivalue_test 2024-08-20T22:20:35.3694965Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:35.3750611Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:35.3804606Z inflating: build/bin/memory_format_test 2024-08-20T22:20:35.3858771Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:35.3911788Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:35.3968396Z inflating: build/bin/native_test 2024-08-20T22:20:35.4019788Z inflating: build/bin/operator_name_test 2024-08-20T22:20:35.4071374Z inflating: build/bin/operators_test 2024-08-20T22:20:35.4123649Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:35.4191482Z inflating: build/bin/pow_test 2024-08-20T22:20:35.4249589Z inflating: build/bin/quantized_test 2024-08-20T22:20:35.4300084Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:35.4351927Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:35.4408472Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:35.4468112Z inflating: build/bin/scalar_test 2024-08-20T22:20:35.4520586Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:35.4573207Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:35.4653515Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:35.4707228Z inflating: build/bin/test_parallel 2024-08-20T22:20:35.4710906Z inflating: build/bin/thread_init_test 2024-08-20T22:20:35.4766850Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:35.4826952Z inflating: build/bin/type_test 2024-08-20T22:20:35.4879378Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:35.4881099Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:35.4951031Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:35.5002865Z inflating: build/bin/weakref_test 2024-08-20T22:20:35.5054674Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:35.5106891Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:35.5166903Z inflating: build/bin/IListRef_test 2024-08-20T22:20:35.5273835Z inflating: build/bin/List_test 2024-08-20T22:20:35.5340810Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:35.5463270Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:35.5560445Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:35.5689811Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:35.5793758Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:35.5855202Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:35.5952211Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:35.6003436Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:35.6059804Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:35.6108877Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:35.6412924Z inflating: build/bin/op_registration_test 2024-08-20T22:20:35.6476412Z inflating: build/bin/inline_container_test 2024-08-20T22:20:35.6530311Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:35.6582459Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:35.6642096Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:35.6697017Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:35.6767796Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:35.6826981Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:35.6876860Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:35.6934522Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:35.6985748Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:35.7051392Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:35.7107940Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:35.7158108Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:35.7209712Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:35.7259478Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:35.7311178Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:35.7364300Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:35.7414949Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:35.7475416Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:35.7525182Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:35.7577295Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:35.7591840Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:35.7658062Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:35.7715402Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:35.7779749Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:35.7842333Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:35.8672539Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:35.9255280Z inflating: build/bin/test_jit 2024-08-20T22:20:35.9255863Z creating: .additional_ci_files/ 2024-08-20T22:20:35.9316182Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:35.9558255Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:35.9593682Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:35.9594088Z rm artifacts.zip 2024-08-20T22:20:35.9603272Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:35.9603778Z env: 2024-08-20T22:20:35.9604068Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:35.9604529Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:35.9605013Z ##[endgroup] 2024-08-20T22:20:36.0824336Z ##[group]Run df -H 2024-08-20T22:20:36.0824679Z df -H 2024-08-20T22:20:36.0833535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.0834038Z env: 2024-08-20T22:20:36.0834320Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.0834779Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.0835268Z ##[endgroup] 2024-08-20T22:20:36.0886425Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:36.0886969Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:36.0887449Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:36.0887922Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:36.0888372Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:36.0889616Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:36.0890113Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:36.0890619Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:36.0931346Z Prepare all required actions 2024-08-20T22:20:36.0931955Z Getting action download info 2024-08-20T22:20:36.2189513Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:36.2189983Z with: 2024-08-20T22:20:36.2190254Z env: 2024-08-20T22:20:36.2190543Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2191000Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2191497Z ##[endgroup] 2024-08-20T22:20:36.2224593Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:36.2225034Z with: 2024-08-20T22:20:36.2225297Z name: td_results 2024-08-20T22:20:36.2225613Z s3-bucket: gha-artifacts 2024-08-20T22:20:36.2225958Z region: us-east-1 2024-08-20T22:20:36.2226255Z env: 2024-08-20T22:20:36.2226525Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2226972Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2227448Z ##[endgroup] 2024-08-20T22:20:36.6932556Z (node:52606) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:36.6933388Z 2024-08-20T22:20:36.6933657Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:36.6934380Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:36.6935239Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:36.7450022Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:36.7456181Z Artifact download has finished successfully 2024-08-20T22:20:36.7773932Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:36.7774429Z mkdir -p .additional_ci_files 2024-08-20T22:20:36.7774997Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:36.7784615Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.7785120Z env: 2024-08-20T22:20:36.7785416Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7785948Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7786443Z ##[endgroup] 2024-08-20T22:20:36.7844890Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:36.7861333Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:36.7896193Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:36.7896680Z .github/scripts/parse_ref.py 2024-08-20T22:20:36.7905547Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:36.7905901Z env: 2024-08-20T22:20:36.7906186Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7906648Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7907128Z ##[endgroup] 2024-08-20T22:20:36.8188161Z Prepare all required actions 2024-08-20T22:20:36.8226697Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:36.8227159Z with: 2024-08-20T22:20:36.8227712Z github-token: *** 2024-08-20T22:20:36.8228024Z env: 2024-08-20T22:20:36.8228303Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8228831Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8229321Z ##[endgroup] 2024-08-20T22:20:36.8267349Z ##[group]Run set -eux 2024-08-20T22:20:36.8267706Z set -eux 2024-08-20T22:20:36.8268298Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:36.8276928Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.8277426Z env: 2024-08-20T22:20:36.8277714Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8278174Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8278815Z GITHUB_TOKEN: *** 2024-08-20T22:20:36.8279131Z ##[endgroup] 2024-08-20T22:20:36.8307327Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-01499a1e34cc6562b 2024-08-20T22:20:38.5372602Z setting job-id=29026529192 2024-08-20T22:20:38.5374407Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.5593443Z Prepare all required actions 2024-08-20T22:20:38.5593904Z Getting action download info 2024-08-20T22:20:38.6820376Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:38.6820843Z with: 2024-08-20T22:20:38.6821292Z github-token: *** 2024-08-20T22:20:38.6830083Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:38.6839506Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.6840257Z env: 2024-08-20T22:20:38.6840539Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.6840986Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.6841464Z ##[endgroup] 2024-08-20T22:20:38.6883864Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:38.6884428Z with: 2024-08-20T22:20:38.6884713Z shell: bash 2024-08-20T22:20:38.6885033Z timeout_minutes: 10 2024-08-20T22:20:38.6885365Z max_attempts: 5 2024-08-20T22:20:38.6885693Z retry_wait_seconds: 30 2024-08-20T22:20:38.6886855Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.6888015Z polling_interval_seconds: 1 2024-08-20T22:20:38.6888398Z warning_on_retry: true 2024-08-20T22:20:38.6888751Z continue_on_error: false 2024-08-20T22:20:38.6889098Z env: 2024-08-20T22:20:38.6889384Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.6889830Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.6890471Z GITHUB_TOKEN: *** 2024-08-20T22:20:38.6890794Z ##[endgroup] 2024-08-20T22:20:38.7645549Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:39.0004818Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:39.1456747Z Collecting requests==2.27.1 2024-08-20T22:20:39.1800931Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:39.3677377Z Collecting pyyaml==6.0.1 2024-08-20T22:20:39.3716570Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:39.6933973Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:39.6983138Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:39.7461297Z Collecting certifi>=2017.4.17 2024-08-20T22:20:39.7500690Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:39.7553687Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:39.7557720Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:39.8313223Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:39.9457946Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:40.7428220Z Command completed after 1 attempt(s). 2024-08-20T22:20:40.7485358Z ##[group]Run set -x 2024-08-20T22:20:40.7485697Z set -x 2024-08-20T22:20:40.7485994Z  2024-08-20T22:20:40.7486542Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.7487210Z # in runner workspace 2024-08-20T22:20:40.7487733Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:40.7496665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.7497207Z env: 2024-08-20T22:20:40.7497492Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.7497953Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.7498442Z ##[endgroup] 2024-08-20T22:20:40.7529052Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:40.7767979Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.7768529Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.7769018Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:40.7769425Z  2024-08-20T22:20:40.7769973Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.7770648Z # in runner workspace 2024-08-20T22:20:40.7771231Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:40.7771879Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:40.7772345Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:40.7781438Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:40.7790898Z  --selected-test-configs "" \ 2024-08-20T22:20:40.7791373Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:40.7791795Z  --tag "${TAG}" \ 2024-08-20T22:20:40.7792197Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:40.7792658Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:40.7793095Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:40.7801892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.7802396Z env: 2024-08-20T22:20:40.7802687Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.7803142Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.7803810Z GITHUB_TOKEN: *** 2024-08-20T22:20:40.7804495Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.7805255Z PR_NUMBER: 2024-08-20T22:20:40.7805573Z TAG: ciflow/inductor/133712 2024-08-20T22:20:40.7805942Z EVENT_NAME: push 2024-08-20T22:20:40.7806253Z SCHEDULE: 2024-08-20T22:20:40.7806577Z HEAD_BRANCH: 2024-08-20T22:20:40.7806897Z ##[endgroup] 2024-08-20T22:20:40.7838254Z Workflow: inductor 2024-08-20T22:20:40.7839242Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.0273362Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:41.1753032Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:41.1753509Z echo "Filtered matrix:" 2024-08-20T22:20:41.1762604Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:41.1771854Z  2024-08-20T22:20:41.1772140Z echo 2024-08-20T22:20:41.1772533Z echo "Is the current job unstable? False" 2024-08-20T22:20:41.1772995Z  2024-08-20T22:20:41.1773279Z echo 2024-08-20T22:20:41.1773647Z echo "Is keep-going label set? False" 2024-08-20T22:20:41.1774096Z  2024-08-20T22:20:41.1774379Z echo 2024-08-20T22:20:41.1774707Z echo "Renabled issues? " 2024-08-20T22:20:41.1783549Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.1784063Z env: 2024-08-20T22:20:41.1784357Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.1784818Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.1785310Z ##[endgroup] 2024-08-20T22:20:41.1815068Z Filtered matrix: 2024-08-20T22:20:41.1826497Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:41.1835404Z 2024-08-20T22:20:41.1835578Z Is the current job unstable? False 2024-08-20T22:20:41.1835859Z 2024-08-20T22:20:41.1836155Z Is keep-going label set? False 2024-08-20T22:20:41.1836423Z 2024-08-20T22:20:41.1836552Z Renabled issues? 2024-08-20T22:20:41.1891332Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.1892079Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.1901152Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.1901656Z env: 2024-08-20T22:20:41.1901949Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.1902409Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.1902910Z JOB_TIMEOUT: 240 2024-08-20T22:20:41.1903222Z ##[endgroup] 2024-08-20T22:20:41.1985466Z ##[group]Run set -x 2024-08-20T22:20:41.1985858Z set -x 2024-08-20T22:20:41.1986155Z  2024-08-20T22:20:41.1986504Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:41.1987039Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:41.1987595Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:41.1988096Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:41.1988504Z else 2024-08-20T22:20:41.1988913Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.1989333Z fi 2024-08-20T22:20:41.1989610Z  2024-08-20T22:20:41.1990071Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:41.1990808Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:41.1991478Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:41.1992064Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:41.1992532Z container_name=$(docker run \ 2024-08-20T22:20:41.1992949Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:41.1993319Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:41.1993717Z  -e PR_NUMBER \ 2024-08-20T22:20:41.1994072Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:41.1994460Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:41.1994864Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:41.1995250Z  -e GITHUB_JOB \ 2024-08-20T22:20:41.1995609Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:41.1995996Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:41.1996399Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:41.1996798Z  -e JOB_ID \ 2024-08-20T22:20:41.1997131Z  -e JOB_NAME \ 2024-08-20T22:20:41.1997479Z  -e BASE_SHA \ 2024-08-20T22:20:41.1997828Z  -e BRANCH \ 2024-08-20T22:20:41.1998155Z  -e SHA1 \ 2024-08-20T22:20:41.1998506Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:41.1998917Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:41.1999294Z  -e SHARD_NUMBER \ 2024-08-20T22:20:41.1999664Z  -e TEST_CONFIG \ 2024-08-20T22:20:41.2000034Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:41.2000425Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:41.2000842Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:41.2001260Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:41.2001659Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:41.2002050Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:41.2002426Z  -e NO_TD \ 2024-08-20T22:20:41.2002764Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:41.2003146Z  -e PR_LABELS \ 2024-08-20T22:20:41.2003554Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:41.2004002Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:41.2004395Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:41.2004804Z  -e XLA_CUDA \ 2024-08-20T22:20:41.2005213Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:41.2005714Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:41.2006229Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:41.2006779Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:41.2007257Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:41.2007701Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:41.2008142Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:41.2008615Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:41.2009169Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:41.2009626Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:41.2010014Z  --ipc=host \ 2024-08-20T22:20:41.2010372Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:41.2010766Z  --tty \ 2024-08-20T22:20:41.2011075Z  --detach \ 2024-08-20T22:20:41.2011574Z  --name="${container_name}" \ 2024-08-20T22:20:41.2012120Z  --user jenkins \ 2024-08-20T22:20:41.2012597Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:41.2013142Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:41.2013571Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:41.2013914Z ) 2024-08-20T22:20:41.2014309Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:41.2015209Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:41.2016153Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:41.2017304Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:41.2025726Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:41.2026077Z env: 2024-08-20T22:20:41.2026364Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.2026841Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.2027445Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:41.2027943Z PR_NUMBER: 2024-08-20T22:20:41.2028274Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:41.2028753Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:41.2029114Z GITHUB_JOB: test 2024-08-20T22:20:41.2029431Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:41.2029801Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:41.2030165Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:41.2030502Z JOB_ID: 29026529192 2024-08-20T22:20:41.2031191Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.2031947Z BRANCH: 2024-08-20T22:20:41.2032292Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.2032814Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.2033291Z TEST_CONFIG: dynamic_inductor_timm 2024-08-20T22:20:41.2033702Z SHARD_NUMBER: 2 2024-08-20T22:20:41.2034027Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:41.2034360Z REENABLED_ISSUES: 2024-08-20T22:20:41.2034698Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:41.2035088Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:41.2035450Z TEST_SHOWLOCALS: False 2024-08-20T22:20:41.2035800Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:41.2036141Z NO_TD: False 2024-08-20T22:20:41.2036452Z TD_DISTRIBUTED: False 2024-08-20T22:20:41.2036870Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:41.2037359Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:41.2037747Z SHM_SIZE: 2g 2024-08-20T22:20:41.2038832Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:41.2039996Z XLA_CUDA: 2024-08-20T22:20:41.2040473Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:41.2041094Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:41.2041538Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:41.2041953Z DASHBOARD_TAG: 2024-08-20T22:20:41.2042470Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:41.2042849Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:41.2043228Z ##[endgroup] 2024-08-20T22:20:41.2074655Z + [[ dynamic_inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:41.2075450Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:41.2075973Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.2083704Z +++ nproc --ignore=2 2024-08-20T22:20:41.2119073Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:52.6509419Z + container_name=ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-20T22:20:52.6515523Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:52.6518727Z + docker exec -i ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:52.7890374Z + echo DOCKER_CONTAINER_ID=ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-20T22:20:52.7894951Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:52.7898008Z + docker exec -t ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:53.1843703Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:53.5102384Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:53.5106407Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:53.5108820Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:53.5111791Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:53.5114873Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:53.5119968Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:53.5136272Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:53.5149191Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:53.5166454Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:53.5552447Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:20:54.1356757Z Installing collected packages: torch 2024-08-20T22:21:04.0203004Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:04.0204537Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:04.0206426Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:04.1036340Z + export TERM=vt100 2024-08-20T22:21:04.1036719Z + TERM=vt100 2024-08-20T22:21:04.1039157Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:04.1049990Z + source .ci/pytorch/common.sh 2024-08-20T22:21:04.1053465Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:04.1062515Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:04.1064554Z +++ declare -f -t trap_add 2024-08-20T22:21:04.1070916Z ++ set -ex 2024-08-20T22:21:04.1071590Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:04.1072243Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:04.1072794Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:04.1075188Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:04.1094507Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:04.1095076Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:04.1095604Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:04.1096007Z + shift 2024-08-20T22:21:04.1096795Z + for trap_add_name in "$@" 2024-08-20T22:21:04.1102449Z +++ trap -p EXIT 2024-08-20T22:21:04.1105175Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:04.1105673Z +++ extract_trap_cmd 2024-08-20T22:21:04.1106163Z +++ printf '%s\n' '' 2024-08-20T22:21:04.1106624Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:04.1108328Z + trap -- ' 2024-08-20T22:21:04.1108876Z cleanup_workspace' EXIT 2024-08-20T22:21:04.1109468Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:04.8199514Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:04.8221376Z + echo 'Environment variables:' 2024-08-20T22:21:04.8221928Z Environment variables: 2024-08-20T22:21:04.8222354Z + env 2024-08-20T22:21:04.8233060Z INSTALLED_DB=yes 2024-08-20T22:21:04.8233837Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.8234365Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.8234811Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.8235449Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.8236163Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.8236845Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.8237448Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.8238032Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.8238539Z HOSTNAME=ef5eb1aef7f0 2024-08-20T22:21:04.8239401Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8240382Z GITHUB_ACTION=__self 2024-08-20T22:21:04.8240842Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.8244868Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.8248743Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.8249252Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.8249604Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.8249954Z TEST_CONFIG=dynamic_inductor_timm 2024-08-20T22:21:04.8250356Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.8250829Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.8251287Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.8251728Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.8252132Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.8252888Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.8253333Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.8253820Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.8254276Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.8254595Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.8254962Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.8255345Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8255781Z XLA_CUDA= 2024-08-20T22:21:04.8256280Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.8259312Z *** 2024-08-20T22:21:04.8259620Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.8260022Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.8260388Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.8260724Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.8261247Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.8261787Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.8262269Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8262791Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.8263278Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8264066Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.8264777Z UCC_HOME=/usr 2024-08-20T22:21:04.8265126Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.8265537Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.8265890Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.8266354Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.8266857Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.8267337Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.8267723Z SHARD_NUMBER=2 2024-08-20T22:21:04.8268036Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.8268396Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.8268880Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.8269319Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.8269800Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.8270281Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.8270648Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.8271074Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.8271517Z NUM_TEST_SHARDS=2 2024-08-20T22:21:04.8271829Z UCX_HOME=/usr 2024-08-20T22:21:04.8272350Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.8273423Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8274689Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.8275941Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8277054Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.8277705Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.8278047Z DASHBOARD_TAG= 2024-08-20T22:21:04.8278363Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.8278834Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.8279374Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.8280370Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8281277Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.8281724Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.8282143Z PR_NUMBER= 2024-08-20T22:21:04.8282431Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.8282768Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.8283210Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.8283657Z TERM=vt100 2024-08-20T22:21:04.8284002Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.8284413Z INSTALLED_VISION=yes 2024-08-20T22:21:04.8284719Z BRANCH= 2024-08-20T22:21:04.8285128Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.8285608Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.8286031Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.8286787Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.8287536Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.8288026Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.8288493Z REENABLED_ISSUES= 2024-08-20T22:21:04.8288793Z SHLVL=1 2024-08-20T22:21:04.8289059Z MAX_JOBS=14 2024-08-20T22:21:04.8289380Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.8289730Z NVARCH=x86_64 2024-08-20T22:21:04.8290033Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.8290491Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8291027Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.8291515Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.8292179Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.8292771Z GITHUB_JOB=test 2024-08-20T22:21:04.8293177Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.8293725Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.8294237Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.8294572Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.8294957Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.8295394Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.8295822Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.8296194Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.8296548Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.8296904Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.8297939Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.8298994Z GITHUB_BASE_REF= 2024-08-20T22:21:04.8299321Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.8299693Z CI=true 2024-08-20T22:21:04.8300028Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.8300446Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.8300819Z JOB_ID=29026529192 2024-08-20T22:21:04.8301140Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.8301475Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.8301807Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.8302273Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.8302750Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.8303093Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.8303451Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.8304366Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8305180Z NO_TD=False 2024-08-20T22:21:04.8305489Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.8305852Z _=/usr/bin/env 2024-08-20T22:21:04.8306337Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:04.8465868Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:04.8466909Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:04.8467800Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:04.8468754Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:04.8469376Z + BUILD_DIR=build 2024-08-20T22:21:04.8469714Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:04.8470112Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:04.8470461Z + SHARD_NUMBER=2 2024-08-20T22:21:04.8470777Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:04.8471106Z + export VALGRIND=ON 2024-08-20T22:21:04.8471431Z + VALGRIND=ON 2024-08-20T22:21:04.8471909Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:04.8472408Z + [[ 0 == \1 ]] 2024-08-20T22:21:04.8472708Z + [[ False == \1 ]] 2024-08-20T22:21:04.8473198Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:04.8473724Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:04.8487080Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:04.8487790Z + [[ -n '' ]] 2024-08-20T22:21:04.8488154Z + echo 'Environment variables' 2024-08-20T22:21:04.8488544Z Environment variables 2024-08-20T22:21:04.8488871Z + env 2024-08-20T22:21:04.8497055Z INSTALLED_DB=yes 2024-08-20T22:21:04.8497441Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.8497931Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.8498472Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.8499249Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.8499972Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.8500616Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.8501293Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.8502032Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.8502680Z HOSTNAME=ef5eb1aef7f0 2024-08-20T22:21:04.8503848Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8504867Z GITHUB_ACTION=__self 2024-08-20T22:21:04.8505304Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.8510105Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.8514468Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.8515115Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.8515478Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.8515889Z TEST_CONFIG=dynamic_inductor_timm 2024-08-20T22:21:04.8516304Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.8516783Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.8517542Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.8517988Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.8518395Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.8518787Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.8519236Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.8519737Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.8520192Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.8520517Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.8520896Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.8521291Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8521739Z XLA_CUDA= 2024-08-20T22:21:04.8522276Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.8522671Z *** 2024-08-20T22:21:04.8522956Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.8523352Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.8523722Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.8524057Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.8524557Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.8525090Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.8525563Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8526080Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.8526568Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8527349Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.8528064Z UCC_HOME=/usr 2024-08-20T22:21:04.8528409Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.8528815Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.8529491Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.8529968Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.8530468Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.8530935Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.8531316Z SHARD_NUMBER=2 2024-08-20T22:21:04.8531629Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.8531984Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.8532343Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.8532785Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.8533266Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.8533749Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.8534115Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.8534546Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.8534986Z NUM_TEST_SHARDS=2 2024-08-20T22:21:04.8535285Z UCX_HOME=/usr 2024-08-20T22:21:04.8535806Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.8536892Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8538163Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.8539412Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8540514Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.8541172Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.8541508Z DASHBOARD_TAG= 2024-08-20T22:21:04.8541809Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.8542278Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.8542819Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.8543806Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8544715Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.8545138Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.8545515Z PR_NUMBER= 2024-08-20T22:21:04.8545805Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.8546120Z VALGRIND=ON 2024-08-20T22:21:04.8546424Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.8546868Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.8547325Z TERM=vt100 2024-08-20T22:21:04.8547661Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.8548077Z INSTALLED_VISION=yes 2024-08-20T22:21:04.8548396Z BRANCH= 2024-08-20T22:21:04.8548749Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.8549149Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.8549571Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.8550327Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.8551067Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.8551551Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.8552024Z REENABLED_ISSUES= 2024-08-20T22:21:04.8552325Z SHLVL=1 2024-08-20T22:21:04.8552585Z MAX_JOBS=14 2024-08-20T22:21:04.8552934Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.8553290Z NVARCH=x86_64 2024-08-20T22:21:04.8553588Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.8554035Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.8554568Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.8555058Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.8555719Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.8556283Z GITHUB_JOB=test 2024-08-20T22:21:04.8556683Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.8557229Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.8557740Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.8558066Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.8558445Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.8558986Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.8559484Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.8559848Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.8560195Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.8560551Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.8561568Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.8562667Z GITHUB_BASE_REF= 2024-08-20T22:21:04.8562994Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.8563362Z CI=true 2024-08-20T22:21:04.8563673Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.8564085Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.8564458Z JOB_ID=29026529192 2024-08-20T22:21:04.8564766Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.8565101Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.8565409Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.8565844Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.8566305Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.8566647Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.8567007Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.8567901Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_54d0de02-44e5-47ab-a806-3fb3ec5bb159 2024-08-20T22:21:04.8568693Z NO_TD=False 2024-08-20T22:21:04.8568994Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.8569358Z _=/usr/bin/env 2024-08-20T22:21:04.8569697Z + echo 'Testing pytorch' 2024-08-20T22:21:04.8570046Z Testing pytorch 2024-08-20T22:21:04.8570375Z + export LANG=C.UTF-8 2024-08-20T22:21:04.8570714Z + LANG=C.UTF-8 2024-08-20T22:21:04.8571002Z + PR_NUMBER= 2024-08-20T22:21:04.8571349Z + [[ dynamic_inductor_timm == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:04.8571864Z + [[ dynamic_inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:04.8572365Z + [[ dynamic_inductor_timm == \s\l\o\w ]] 2024-08-20T22:21:04.8572990Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:04.8573691Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:04.8574240Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.8574715Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.8575161Z + [[ dynamic_inductor_timm == *crossref* ]] 2024-08-20T22:21:04.8575745Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:04.8576378Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:04.8577027Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:04.8577587Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:04.8578125Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:06.0381730Z Collecting ninja==1.10.2 2024-08-20T22:21:06.0566432Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:06.1121944Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:06.7106870Z Installing collected packages: ninja 2024-08-20T22:21:06.7185642Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:06.7186915Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:06.7756006Z Successfully installed ninja-1.10.2 2024-08-20T22:21:06.8538242Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.8540329Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.8541747Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:06.8542253Z + install_tlparse 2024-08-20T22:21:06.8542644Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:06.8543590Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:07.2840504Z Collecting tlparse==0.3.25 2024-08-20T22:21:07.3029451Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:07.3541578Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:08.0047901Z Installing collected packages: tlparse 2024-08-20T22:21:08.0389092Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:08.1165163Z ++ python -m site --user-base 2024-08-20T22:21:08.1407968Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:08.1409570Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:08.1410233Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:08.1410936Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:08.1411886Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:08.1413077Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:08.1413779Z + cd test 2024-08-20T22:21:08.1414315Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:09.7664564Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:09.7665202Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:09.7668201Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:09.7668838Z + [[ dynamic_inductor_timm == *pr_time_benchmarks* ]] 2024-08-20T22:21:09.7669439Z + [[ dynamic_inductor_timm == *dynamo_eager* ]] 2024-08-20T22:21:09.7669998Z + [[ dynamic_inductor_timm == *aot_eager* ]] 2024-08-20T22:21:09.7670490Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-08-20T22:21:09.7671075Z + [[ dynamic_inductor_timm == *inductor* ]] 2024-08-20T22:21:09.7671526Z + [[ dynamic_inductor_timm != *perf* ]] 2024-08-20T22:21:09.7672186Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:09.7672627Z + [[ dynamic_inductor_timm == *dynamic* ]] 2024-08-20T22:21:09.7673254Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-08-20T22:21:09.7673812Z + [[ dynamic_inductor_timm == *cpu* ]] 2024-08-20T22:21:09.7674296Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:09.7703121Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:09.7703809Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:09.7706482Z + cd test 2024-08-20T22:21:09.7706955Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:11.2241593Z PyTorch built with: 2024-08-20T22:21:11.2242190Z - GCC 9.4 2024-08-20T22:21:11.2242578Z - C++ Version: 201703 2024-08-20T22:21:11.2243458Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:11.2244552Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:11.2245240Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:11.2245763Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:11.2246248Z - NNPACK is enabled 2024-08-20T22:21:11.2246632Z - CPU capability usage: AVX2 2024-08-20T22:21:11.2247046Z - CUDA Runtime 12.1 2024-08-20T22:21:11.2247572Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:11.2248181Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:11.2248622Z - Magma 2.6.1 2024-08-20T22:21:11.2255627Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:11.2262013Z 2024-08-20T22:21:11.5430081Z + cd test 2024-08-20T22:21:11.5430863Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:12.8862483Z ATen/Parallel: 2024-08-20T22:21:12.8862886Z at::get_num_threads() : 8 2024-08-20T22:21:12.8863295Z at::get_num_interop_threads() : 16 2024-08-20T22:21:12.8863814Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:12.8864238Z omp_get_max_threads() : 8 2024-08-20T22:21:12.8865227Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:12.8866017Z mkl_get_max_threads() : 8 2024-08-20T22:21:12.8866645Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:12.8867272Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:12.8867699Z Environment variables: 2024-08-20T22:21:12.8868036Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:12.8868400Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:12.8868847Z ATen parallel backend: OpenMP 2024-08-20T22:21:12.8869110Z 2024-08-20T22:21:13.1600735Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:13.1601439Z + [[ dynamic_inductor_timm == *backward* ]] 2024-08-20T22:21:13.1602020Z + [[ dynamic_inductor_timm == *xla* ]] 2024-08-20T22:21:13.1602566Z + [[ dynamic_inductor_timm == *executorch* ]] 2024-08-20T22:21:13.1603080Z + [[ dynamic_inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:13.1603726Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:13.1604286Z + [[ dynamic_inductor_timm == distributed ]] 2024-08-20T22:21:13.1604812Z + [[ dynamic_inductor_timm == *inductor_distributed* ]] 2024-08-20T22:21:13.1605406Z + [[ dynamic_inductor_timm == *inductor-halide* ]] 2024-08-20T22:21:13.1606025Z + [[ dynamic_inductor_timm == *inductor-micro-benchmark* ]] 2024-08-20T22:21:13.1606668Z + [[ dynamic_inductor_timm == *huggingface* ]] 2024-08-20T22:21:13.1607152Z + [[ dynamic_inductor_timm == *timm* ]] 2024-08-20T22:21:13.1607576Z + install_torchvision 2024-08-20T22:21:13.1607922Z + local orig_preload 2024-08-20T22:21:13.1608255Z + local commit 2024-08-20T22:21:13.1608573Z ++ get_pinned_commit vision 2024-08-20T22:21:13.1608967Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:13.1624636Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:13.1625167Z + orig_preload= 2024-08-20T22:21:13.1625568Z + '[' -n '' ']' 2024-08-20T22:21:13.1626398Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:13.1627829Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:13.4963784Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:13.4968656Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-_3yf4h54 2024-08-20T22:21:13.4997101Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-_3yf4h54 2024-08-20T22:21:14.9603093Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:14.9635107Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:16.2863351Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:16.5939102Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:19.1068816Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:19.1100532Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:19.1104137Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:19.1108396Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:19.1181510Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:19.1185430Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:19.1187301Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:19.1189034Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:19.1192111Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:19.1196975Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:19.1210941Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:19.1685359Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:19.1748053Z Building wheels for collected packages: torchvision 2024-08-20T22:22:35.7517314Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:22:35.7552215Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116122 sha256=46a42bf875b2ffe35e9a7c76f1d463921ebf92ece1764d7c6507a98e6a74ebd0 2024-08-20T22:22:35.7554716Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:35.7587904Z Successfully built torchvision 2024-08-20T22:22:36.2413409Z Installing collected packages: torchvision 2024-08-20T22:22:36.6524147Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:36.7925055Z + '[' -n '' ']' 2024-08-20T22:22:36.7927155Z + id=1 2024-08-20T22:22:36.7927640Z + test_dynamo_benchmark timm_models 1 2024-08-20T22:22:36.7928348Z ++ pwd 2024-08-20T22:22:36.7933042Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:36.7933675Z + local suite=timm_models 2024-08-20T22:22:36.7934024Z + shift 2024-08-20T22:22:36.7934310Z + local shard_id=1 2024-08-20T22:22:36.7937506Z + shift 2024-08-20T22:22:36.7938272Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-08-20T22:22:36.7938977Z + [[ dynamic_inductor_timm == *perf* ]] 2024-08-20T22:22:36.7939421Z + [[ dynamic_inductor_timm == *cpu* ]] 2024-08-20T22:22:36.7939887Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-08-20T22:22:36.7940662Z + test_single_dynamo_benchmark inference timm_models 1 --inference --bfloat16 2024-08-20T22:22:36.7941242Z ++ pwd 2024-08-20T22:22:36.7941769Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:36.7942452Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:36.7967873Z + local name=inference 2024-08-20T22:22:36.7968292Z + shift 2024-08-20T22:22:36.7968654Z + local suite=timm_models 2024-08-20T22:22:36.7969096Z + shift 2024-08-20T22:22:36.7969463Z + local shard_id=1 2024-08-20T22:22:36.7969848Z + shift 2024-08-20T22:22:36.7970131Z + partition_flags=() 2024-08-20T22:22:36.7970470Z + local partition_flags 2024-08-20T22:22:36.7970875Z + [[ -n 2 ]] 2024-08-20T22:22:36.7971188Z + [[ -n 1 ]] 2024-08-20T22:22:36.7971809Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:22:36.7972504Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-08-20T22:22:36.7972994Z + [[ dynamic_inductor_timm == *perf* ]] 2024-08-20T22:22:36.7973458Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-08-20T22:22:36.7973927Z + [[ dynamic_inductor_timm == *_avx2* ]] 2024-08-20T22:22:36.7974387Z + [[ dynamic_inductor_timm == *_avx512* ]] 2024-08-20T22:22:36.7976176Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-08-20T22:22:43.1925381Z 2024-08-20T22:22:43.3394929Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:22:43.3395233Z 2024-08-20T22:22:43.4219389Z model.safetensors: 0% 0.00/29.6M [00:00 will be ignored 2024-08-20T22:49:44.2383751Z pass 2024-08-20T22:49:44.2430362Z TIMING: entire_frame_compile:98.34788 code_gen:29.53219 inductor_compile:54.6327 backend_compile:79.95306 total_wall_time:112.46876 2024-08-20T22:49:44.2433000Z STATS: call_* op count: 700 | FakeTensorMode.__torch_dispatch__:50772 | attempt fast:1800 | fast is_contiguous:1800 | FakeTensor.__torch_dispatch__:16116 | ProxyTorchDispatchMode.__torch_dispatch__:17697 2024-08-20T22:49:44.2434336Z Dynamo produced 4 graphs covering 700 ops with 7 graph breaks (5 unique) 2024-08-20T22:49:52.5632779Z 2024-08-20T22:49:53.4207440Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:49:53.4207951Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:49:53.4208381Z cuda train mobilenetv2_100 2024-08-20T22:51:05.7628175Z W0820 22:51:05.760000 15460 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:51:46.0119892Z pass 2024-08-20T22:51:46.0167078Z TIMING: entire_frame_compile:106.22545 code_gen:27.78694 inductor_compile:54.61687 backend_compile:84.79069 total_wall_time:122.04519 2024-08-20T22:51:46.0169023Z STATS: call_* op count: 699 | FakeTensorMode.__torch_dispatch__:56464 | attempt fast:1936 | fast is_contiguous:1936 | FakeTensor.__torch_dispatch__:17194 | ProxyTorchDispatchMode.__torch_dispatch__:17713 2024-08-20T22:51:46.0170513Z Dynamo produced 3 graphs covering 699 ops with 7 graph breaks (5 unique) 2024-08-20T22:51:54.1301849Z 2024-08-20T22:51:55.0403836Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:55.0404327Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:55.0404783Z cuda train mobilenetv3_large_100 2024-08-20T22:53:15.7556752Z W0820 22:53:15.754000 16140 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:54:02.1163909Z pass 2024-08-20T22:54:02.1234272Z TIMING: entire_frame_compile:119.4037 code_gen:33.12494 inductor_compile:63.47211 backend_compile:96.18565 total_wall_time:139.38335 2024-08-20T22:54:02.1235870Z STATS: call_* op count: 775 | FakeTensorMode.__torch_dispatch__:60706 | attempt fast:2386 | fast is_contiguous:2386 | FakeTensor.__torch_dispatch__:18316 | ProxyTorchDispatchMode.__torch_dispatch__:19569 2024-08-20T22:54:02.1237174Z Dynamo produced 3 graphs covering 775 ops with 7 graph breaks (5 unique) 2024-08-20T22:54:11.4514789Z 2024-08-20T22:54:12.3619181Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:12.3620082Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:12.3620935Z cuda train mobilevit_s 2024-08-20T22:56:46.8265141Z pass 2024-08-20T22:56:46.8350176Z TIMING: entire_frame_compile:139.29632 code_gen:55.42675 inductor_compile:88.33888 backend_compile:120.96774 total_wall_time:195.92882 2024-08-20T22:56:46.8354411Z STATS: call_* op count: 370 | FakeTensorMode.__torch_dispatch__:59182 | attempt fast:3155 | fast is_contiguous:3143 | FakeTensor.__torch_dispatch__:10338 | ProxyTorchDispatchMode.__torch_dispatch__:16472 | slow no contiguity match:12 2024-08-20T22:56:46.8355881Z Dynamo produced 3 graphs covering 370 ops with 6 graph breaks (5 unique) 2024-08-20T22:57:00.2909317Z 2024-08-20T22:57:01.5418554Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:57:01.5419180Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:57:01.5419715Z cuda train nfnet_l0 2024-08-20T22:58:17.5438216Z W0820 22:58:17.542000 18392 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:59:19.4872476Z pass 2024-08-20T22:59:19.5109146Z TIMING: entire_frame_compile:128.79539 code_gen:35.07338 inductor_compile:64.63495 backend_compile:102.51449 total_wall_time:149.75124 2024-08-20T22:59:19.5112145Z STATS: call_* op count: 1205 | FakeTensorMode.__torch_dispatch__:72964 | attempt fast:1242 | fast is_contiguous:1242 | FakeTensor.__torch_dispatch__:20762 | ProxyTorchDispatchMode.__torch_dispatch__:25953 2024-08-20T22:59:19.5113481Z Dynamo produced 3 graphs covering 1205 ops with 7 graph breaks (5 unique) 2024-08-20T22:59:28.9712747Z 2024-08-20T22:59:30.1345139Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:59:30.1347320Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:59:30.1347805Z cuda train pit_b_224 2024-08-20T23:00:39.9336767Z pass 2024-08-20T23:00:39.9831022Z TIMING: entire_frame_compile:61.65809 code_gen:13.00682 inductor_compile:29.74576 backend_compile:49.56917 total_wall_time:79.31199 2024-08-20T23:00:39.9834523Z STATS: call_* op count: 310 | FakeTensorMode.__torch_dispatch__:44255 | attempt fast:2576 | fast is_contiguous:2576 | FakeTensor.__torch_dispatch__:6182 | ProxyTorchDispatchMode.__torch_dispatch__:12561 2024-08-20T23:00:39.9835861Z Dynamo produced 2 graphs covering 310 ops with 6 graph breaks (5 unique) 2024-08-20T23:00:48.1665606Z 2024-08-20T23:00:49.9904149Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:00:49.9904768Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:00:49.9907935Z cuda train pnasnet5large 2024-08-20T23:10:13.9249945Z pass 2024-08-20T23:10:14.0680616Z TIMING: entire_frame_compile:498.6411 code_gen:181.29677 inductor_compile:306.74108 backend_compile:442.10034 total_wall_time:709.42376 2024-08-20T23:10:14.0682360Z STATS: call_* op count: 1161 | FakeTensorMode.__torch_dispatch__:176491 | attempt fast:8233 | fast is_contiguous:8185 | FakeTensor.__torch_dispatch__:34392 | ProxyTorchDispatchMode.__torch_dispatch__:47606 | slow no contiguity match:48 2024-08-20T23:10:14.0683814Z Dynamo produced 2 graphs covering 1161 ops with 5 graph breaks (4 unique) 2024-08-20T23:11:24.8128152Z 2024-08-20T23:11:26.2493109Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:11:26.2493595Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:11:26.2495809Z cuda train poolformer_m36 2024-08-20T23:15:44.6763584Z pass 2024-08-20T23:15:44.7181231Z TIMING: entire_frame_compile:247.68673 code_gen:141.57156 inductor_compile:181.18492 backend_compile:226.27368 total_wall_time:391.73934 2024-08-20T23:15:44.7183952Z STATS: call_* op count: 554 | FakeTensorMode.__torch_dispatch__:78780 | attempt fast:7695 | fast is_contiguous:7551 | ProxyTorchDispatchMode.__torch_dispatch__:22827 | slow both tensors nontrivially broadcast:144 | FakeTensor.__torch_dispatch__:15793 2024-08-20T23:15:44.7185499Z Dynamo produced 2 graphs covering 554 ops with 6 graph breaks (5 unique) 2024-08-20T23:15:55.4646355Z 2024-08-20T23:15:56.3486691Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:15:56.3487181Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:15:56.3487613Z cuda train regnety_002 2024-08-20T23:17:07.3546787Z pass 2024-08-20T23:17:07.3583318Z TIMING: entire_frame_compile:66.5293 code_gen:15.46462 inductor_compile:33.56336 backend_compile:53.91697 total_wall_time:82.88336 2024-08-20T23:17:07.3587220Z STATS: call_* op count: 271 | FakeTensorMode.__torch_dispatch__:41345 | attempt fast:1826 | fast is_contiguous:1826 | FakeTensor.__torch_dispatch__:8391 | ProxyTorchDispatchMode.__torch_dispatch__:12209 2024-08-20T23:17:07.3588726Z Dynamo produced 2 graphs covering 271 ops with 6 graph breaks (5 unique) 2024-08-20T23:17:14.5637112Z 2024-08-20T23:17:15.7155530Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:17:15.7156422Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:17:15.7157307Z cuda train repvgg_a2 2024-08-20T23:18:20.2300821Z W0820 23:18:20.228000 22868 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:19:04.1686903Z pass 2024-08-20T23:19:04.1873716Z TIMING: entire_frame_compile:100.30116 code_gen:30.14907 inductor_compile:54.84188 backend_compile:80.70654 total_wall_time:114.98688 2024-08-20T23:19:04.1875536Z STATS: call_* op count: 756 | FakeTensorMode.__torch_dispatch__:53087 | attempt fast:2269 | fast is_contiguous:2269 | FakeTensor.__torch_dispatch__:17182 | ProxyTorchDispatchMode.__torch_dispatch__:18975 2024-08-20T23:19:04.1876850Z Dynamo produced 3 graphs covering 756 ops with 7 graph breaks (5 unique) 2024-08-20T23:19:12.4805549Z 2024-08-20T23:19:14.0238271Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:19:14.0238736Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:19:14.0244842Z cuda train res2net101_26w_4s 2024-08-20T23:22:57.9954119Z pass 2024-08-20T23:22:58.0301002Z TIMING: entire_frame_compile:209.38665 code_gen:51.39862 inductor_compile:109.51055 backend_compile:175.8061 total_wall_time:268.32108 2024-08-20T23:22:58.0304105Z STATS: call_* op count: 977 | FakeTensorMode.__torch_dispatch__:135592 | attempt fast:6747 | fast is_contiguous:6341 | FakeTensor.__torch_dispatch__:30362 | slow no contiguity match:406 | ProxyTorchDispatchMode.__torch_dispatch__:37834 2024-08-20T23:22:58.0305595Z Dynamo produced 2 graphs covering 977 ops with 6 graph breaks (5 unique) 2024-08-20T23:23:12.5603414Z 2024-08-20T23:23:13.8305149Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:13.8305649Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:23:13.8306090Z cuda train res2net50_14w_8s 2024-08-20T23:26:55.3457377Z pass 2024-08-20T23:26:55.3664951Z TIMING: entire_frame_compile:206.10081 code_gen:49.5921 inductor_compile:109.61565 backend_compile:175.6584 total_wall_time:266.8295 2024-08-20T23:26:55.3667938Z STATS: call_* op count: 852 | FakeTensorMode.__torch_dispatch__:125326 | attempt fast:5939 | fast is_contiguous:5435 | FakeTensor.__torch_dispatch__:26231 | slow no contiguity match:504 | ProxyTorchDispatchMode.__torch_dispatch__:32715 2024-08-20T23:26:55.3669538Z Dynamo produced 2 graphs covering 852 ops with 6 graph breaks (5 unique) 2024-08-20T23:27:13.1764631Z 2024-08-20T23:27:14.2946620Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:27:14.2947102Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:27:14.2947536Z cuda train res2next50 2024-08-20T23:29:11.8524777Z pass 2024-08-20T23:29:11.8702927Z TIMING: entire_frame_compile:109.99288 code_gen:30.49834 inductor_compile:60.95581 backend_compile:92.89262 total_wall_time:142.93863 2024-08-20T23:29:11.8707000Z STATS: call_* op count: 484 | FakeTensorMode.__torch_dispatch__:67437 | attempt fast:3347 | fast is_contiguous:3179 | FakeTensor.__torch_dispatch__:15105 | slow no contiguity match:168 | ProxyTorchDispatchMode.__torch_dispatch__:18879 2024-08-20T23:29:11.8710071Z Dynamo produced 2 graphs covering 484 ops with 6 graph breaks (5 unique) 2024-08-20T23:29:21.0253989Z 2024-08-20T23:29:21.7505191Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:21.7505767Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:21.7508814Z cuda train resmlp_12_224 2024-08-20T23:30:02.9257042Z pass 2024-08-20T23:30:02.9384645Z TIMING: entire_frame_compile:37.54424 code_gen:8.50527 inductor_compile:17.04744 backend_compile:30.33258 total_wall_time:48.19756 2024-08-20T23:30:02.9390412Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:27009 | attempt fast:1512 | slow no contiguity match:552 | fast is_contiguous:960 | ProxyTorchDispatchMode.__torch_dispatch__:8345 | FakeTensor.__torch_dispatch__:3939 2024-08-20T23:30:02.9392541Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-08-20T23:30:08.5332340Z 2024-08-20T23:30:10.1855881Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:10.1856451Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:30:10.1856963Z cuda train resnest101e 2024-08-20T23:33:59.6609381Z pass 2024-08-20T23:33:59.7023589Z TIMING: entire_frame_compile:212.19911 code_gen:51.1192 inductor_compile:111.59012 backend_compile:178.06661 total_wall_time:273.41315 2024-08-20T23:33:59.7027689Z STATS: call_* op count: 1028 | FakeTensorMode.__torch_dispatch__:132227 | attempt fast:5879 | fast is_contiguous:5747 | FakeTensor.__torch_dispatch__:26591 | ProxyTorchDispatchMode.__torch_dispatch__:38715 | slow no contiguity match:132 2024-08-20T23:33:59.7029282Z Dynamo produced 2 graphs covering 1028 ops with 6 graph breaks (5 unique) 2024-08-20T23:34:16.2504373Z 2024-08-20T23:34:17.3550398Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:34:17.3550875Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:34:17.3556683Z cuda train rexnet_100 2024-08-20T23:36:27.6468749Z W0820 23:36:27.646000 27048 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:37:32.7904879Z pass 2024-08-20T23:37:32.7989187Z TIMING: entire_frame_compile:177.73274 code_gen:50.84857 inductor_compile:94.41712 backend_compile:143.79745 total_wall_time:212.34213 2024-08-20T23:37:32.7992846Z STATS: call_* op count: 1034 | FakeTensorMode.__torch_dispatch__:80701 | attempt fast:2729 | fast is_contiguous:2652 | FakeTensor.__torch_dispatch__:23812 | slow no contiguity match:77 | ProxyTorchDispatchMode.__torch_dispatch__:26051 2024-08-20T23:37:32.7995819Z Dynamo produced 3 graphs covering 1034 ops with 7 graph breaks (5 unique) 2024-08-20T23:37:51.8256897Z 2024-08-20T23:37:52.9449467Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:52.9449926Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:37:52.9452735Z cuda train sebotnet33ts_256 2024-08-20T23:39:52.1119080Z pass 2024-08-20T23:39:52.1245602Z TIMING: entire_frame_compile:104.9116 code_gen:30.28036 inductor_compile:54.86483 backend_compile:88.57648 total_wall_time:133.57481 2024-08-20T23:39:52.1247330Z STATS: call_* op count: 373 | FakeTensorMode.__torch_dispatch__:50431 | attempt fast:2254 | fast is_contiguous:2219 | FakeTensor.__torch_dispatch__:9072 | slow no contiguity match:35 | ProxyTorchDispatchMode.__torch_dispatch__:13430 2024-08-20T23:39:52.1248855Z Dynamo produced 3 graphs covering 373 ops with 6 graph breaks (5 unique) 2024-08-20T23:40:04.7225655Z 2024-08-20T23:40:05.9239652Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:40:05.9240147Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:40:05.9240587Z cuda train selecsls42b 2024-08-20T23:41:01.2036125Z pass 2024-08-20T23:41:01.2307730Z TIMING: entire_frame_compile:50.78441 code_gen:14.56619 inductor_compile:28.00052 backend_compile:42.48819 total_wall_time:64.33456 2024-08-20T23:41:01.2311194Z STATS: call_* op count: 178 | FakeTensorMode.__torch_dispatch__:29436 | attempt fast:1421 | fast is_contiguous:1391 | FakeTensor.__torch_dispatch__:6628 | ProxyTorchDispatchMode.__torch_dispatch__:8917 | slow no contiguity match:30 2024-08-20T23:41:01.2312686Z Dynamo produced 2 graphs covering 178 ops with 6 graph breaks (5 unique) 2024-08-20T23:41:07.7341920Z 2024-08-20T23:41:08.6067906Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:41:08.6068690Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:41:08.6070909Z cuda train spnasnet_100 2024-08-20T23:42:21.8775724Z W0820 23:42:21.876000 30615 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:43:12.7885985Z pass 2024-08-20T23:43:12.7926185Z TIMING: entire_frame_compile:116.23457 code_gen:34.06084 inductor_compile:63.47304 backend_compile:93.78627 total_wall_time:132.77811 2024-08-20T23:43:12.7928875Z STATS: call_* op count: 855 | FakeTensorMode.__torch_dispatch__:62100 | attempt fast:2224 | fast is_contiguous:2224 | FakeTensor.__torch_dispatch__:19748 | ProxyTorchDispatchMode.__torch_dispatch__:21710 2024-08-20T23:43:12.7930211Z Dynamo produced 3 graphs covering 855 ops with 7 graph breaks (5 unique) 2024-08-20T23:43:22.2771799Z 2024-08-20T23:43:24.3124779Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:43:24.3125259Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:43:24.3125717Z cuda train swin_base_patch4_window7_224 2024-08-20T23:47:47.1852898Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:47:47.1855395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-20T23:47:47.1856374Z pred = mod(*cloned_inputs) 2024-08-20T23:47:47.1857805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 838, in forward 2024-08-20T23:47:47.1858843Z x = self.forward_features(x) 2024-08-20T23:47:47.1859975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 830, in forward_features 2024-08-20T23:47:47.1860992Z x = self.layers(x) 2024-08-20T23:47:47.1862072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 559, in forward 2024-08-20T23:47:47.1862871Z x = self.blocks(x) 2024-08-20T23:47:47.1863707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 406, in forward 2024-08-20T23:47:47.1864876Z x = x + self.drop_path1(self._attn(self.norm1(x))) 2024-08-20T23:47:47.1865894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 390, in _attn 2024-08-20T23:47:47.1866923Z attn_windows = self.attn(x_windows, mask=attn_mask) # nW*B, window_size*window_size, C 2024-08-20T23:47:47.1868123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 194, in forward 2024-08-20T23:47:47.1869047Z attn = attn + self._get_rel_pos_bias() 2024-08-20T23:47:47.1870044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 165, in _get_rel_pos_bias 2024-08-20T23:47:47.1871015Z relative_position_bias = self.relative_position_bias_table[ 2024-08-20T23:47:47.1871414Z 2024-08-20T23:48:10.4406389Z W0820 23:48:10.439000 30915 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:49:54.2225800Z pass 2024-08-20T23:49:54.2641383Z TIMING: entire_frame_compile:335.30427 code_gen:68.36179 inductor_compile:142.14499 backend_compile:280.2369 total_wall_time:384.40152 2024-08-20T23:49:54.2643536Z STATS: call_* op count: 2449 | FakeTensorMode.__torch_dispatch__:162693 | attempt fast:6330 | fast is_contiguous:6186 | slow no contiguity match:144 | ProxyTorchDispatchMode.__torch_dispatch__:50142 | FakeTensor.__torch_dispatch__:29777 2024-08-20T23:49:54.2645519Z Dynamo produced 3 graphs covering 2449 ops with 7 graph breaks (5 unique) 2024-08-20T23:51:00.7072224Z 2024-08-20T23:51:03.9277972Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:51:03.9280564Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:51:03.9281044Z cuda train swsl_resnext101_32x16d 2024-08-20T23:53:09.6103355Z pass 2024-08-20T23:53:09.7471308Z TIMING: entire_frame_compile:109.01371 code_gen:22.98235 inductor_compile:52.57585 backend_compile:88.59967 total_wall_time:133.21054 2024-08-20T23:53:09.7474738Z STATS: call_* op count: 453 | FakeTensorMode.__torch_dispatch__:70197 | attempt fast:3635 | fast is_contiguous:3635 | FakeTensor.__torch_dispatch__:16128 | ProxyTorchDispatchMode.__torch_dispatch__:22001 2024-08-20T23:53:09.7476138Z Dynamo produced 2 graphs covering 453 ops with 6 graph breaks (5 unique) 2024-08-20T23:53:19.3514616Z 2024-08-20T23:53:20.3262558Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:20.3263040Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:20.3263489Z cuda train tf_efficientnet_b0 2024-08-20T23:55:01.0175112Z pass 2024-08-20T23:55:01.0260663Z TIMING: entire_frame_compile:94.90502 code_gen:21.58484 inductor_compile:48.55933 backend_compile:77.47827 total_wall_time:120.90861 2024-08-20T23:55:01.0263839Z STATS: call_* op count: 297 | FakeTensorMode.__torch_dispatch__:52500 | attempt fast:2747 | fast is_contiguous:2747 | FakeTensor.__torch_dispatch__:11181 | ProxyTorchDispatchMode.__torch_dispatch__:14649 2024-08-20T23:55:01.0266496Z Dynamo produced 2 graphs covering 297 ops with 6 graph breaks (5 unique) 2024-08-20T23:55:09.6923539Z 2024-08-20T23:55:10.7572601Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:10.7573083Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:55:10.7577452Z cuda train tf_mixnet_l 2024-08-20T23:57:25.5431917Z pass 2024-08-20T23:57:25.5524089Z TIMING: entire_frame_compile:127.59736 code_gen:32.17403 inductor_compile:65.1264 backend_compile:103.26303 total_wall_time:161.01023 2024-08-20T23:57:25.5525856Z STATS: call_* op count: 598 | FakeTensorMode.__torch_dispatch__:75396 | attempt fast:3078 | fast is_contiguous:3078 | FakeTensor.__torch_dispatch__:14179 | ProxyTorchDispatchMode.__torch_dispatch__:19823 2024-08-20T23:57:25.5527177Z Dynamo produced 2 graphs covering 598 ops with 6 graph breaks (5 unique) 2024-08-20T23:57:34.9638429Z 2024-08-20T23:57:35.9516847Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:57:35.9517912Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:57:35.9518512Z cuda train tinynet_a 2024-08-20T23:59:27.6713441Z pass 2024-08-20T23:59:27.6797723Z TIMING: entire_frame_compile:106.11895 code_gen:23.55217 inductor_compile:52.32344 backend_compile:86.50256 total_wall_time:133.91184 2024-08-20T23:59:27.6803170Z STATS: call_* op count: 346 | FakeTensorMode.__torch_dispatch__:61306 | attempt fast:3260 | fast is_contiguous:3260 | FakeTensor.__torch_dispatch__:13215 | ProxyTorchDispatchMode.__torch_dispatch__:17283 2024-08-20T23:59:27.6804818Z Dynamo produced 2 graphs covering 346 ops with 6 graph breaks (5 unique) 2024-08-20T23:59:35.8589477Z 2024-08-20T23:59:36.8788511Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:36.8789237Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:59:36.8789915Z cuda train tnt_s_patch16_224 2024-08-21T00:03:11.2891980Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:03:11.2893159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-21T00:03:11.2897141Z pred = mod(*cloned_inputs) 2024-08-21T00:03:11.2898259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 335, in forward 2024-08-21T00:03:11.2899029Z x = self.forward_features(x) 2024-08-21T00:03:11.2900169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 311, in forward_features 2024-08-21T00:03:11.2901242Z pixel_embed = self.pixel_embed(x, self.pixel_pos) 2024-08-21T00:03:11.2902170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 183, in forward 2024-08-21T00:03:11.2902993Z x = self.unfold(x) 2024-08-21T00:03:11.2903210Z 2024-08-21T00:03:30.8098790Z W0821 00:03:30.808000 34636 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:05:12.3664088Z pass 2024-08-21T00:05:12.3912480Z TIMING: entire_frame_compile:294.16967 code_gen:63.04469 inductor_compile:136.96451 backend_compile:246.39832 total_wall_time:349.57888 2024-08-21T00:05:12.3915352Z STATS: call_* op count: 1850 | FakeTensorMode.__torch_dispatch__:150153 | attempt fast:6402 | fast is_contiguous:6186 | slow no contiguity match:216 | ProxyTorchDispatchMode.__torch_dispatch__:49261 | FakeTensor.__torch_dispatch__:30172 2024-08-21T00:05:12.3916857Z Dynamo produced 3 graphs covering 1850 ops with 7 graph breaks (5 unique) 2024-08-21T00:06:05.6071244Z 2024-08-21T00:06:06.9576104Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:06:06.9579040Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:06:06.9579605Z cuda train twins_pcpvt_base 2024-08-21T00:09:06.5122438Z W0821 00:09:06.511000 35308 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:11:19.8425629Z pass 2024-08-21T00:11:19.8731669Z TIMING: entire_frame_compile:287.66151 code_gen:82.61365 inductor_compile:153.52692 backend_compile:238.01357 total_wall_time:339.78325 2024-08-21T00:11:19.8734819Z STATS: call_* op count: 2432 | FakeTensorMode.__torch_dispatch__:178592 | attempt fast:7156 | fast is_contiguous:7032 | FakeTensor.__torch_dispatch__:43714 | slow no contiguity match:122 | ProxyTorchDispatchMode.__torch_dispatch__:59071 | fast channels_last:2 2024-08-21T00:11:19.8736409Z Dynamo produced 3 graphs covering 2432 ops with 7 graph breaks (5 unique) 2024-08-21T00:11:41.3411220Z 2024-08-21T00:11:42.6088203Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:11:42.6088691Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:11:42.6091034Z cuda train visformer_small 2024-08-21T00:12:44.7868000Z W0821 00:12:44.785000 36392 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:13:18.6505960Z pass 2024-08-21T00:13:18.6759223Z TIMING: entire_frame_compile:88.40485 code_gen:21.62924 inductor_compile:42.81656 backend_compile:70.98956 total_wall_time:101.52123 2024-08-21T00:13:18.6760868Z STATS: call_* op count: 703 | FakeTensorMode.__torch_dispatch__:49364 | attempt fast:2297 | fast is_contiguous:2297 | FakeTensor.__torch_dispatch__:13063 | ProxyTorchDispatchMode.__torch_dispatch__:15868 2024-08-21T00:13:18.6762180Z Dynamo produced 3 graphs covering 703 ops with 7 graph breaks (5 unique) 2024-08-21T00:13:26.7549755Z 2024-08-21T00:13:28.3029606Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:28.3030088Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:13:28.3030527Z cuda train vit_base_patch16_224 2024-08-21T00:14:21.5097247Z W0821 00:14:21.508000 36834 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:14:58.7709426Z pass 2024-08-21T00:14:58.8230023Z TIMING: entire_frame_compile:80.70496 code_gen:22.21195 inductor_compile:42.72918 backend_compile:66.38639 total_wall_time:93.60583 2024-08-21T00:14:58.8232976Z STATS: call_* op count: 740 | FakeTensorMode.__torch_dispatch__:53652 | attempt fast:2319 | fast is_contiguous:2319 | FakeTensor.__torch_dispatch__:12926 | ProxyTorchDispatchMode.__torch_dispatch__:17807 2024-08-21T00:14:58.8234328Z Dynamo produced 3 graphs covering 740 ops with 7 graph breaks (5 unique) 2024-08-21T00:15:06.4276027Z 2024-08-21T00:15:07.6909910Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:15:07.6910386Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:15:07.6913997Z cuda train volo_d1_224 2024-08-21T00:16:31.9917636Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:16:31.9918974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-21T00:16:31.9922674Z pred = mod(*cloned_inputs) 2024-08-21T00:16:31.9923888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 822, in forward 2024-08-21T00:16:31.9924671Z x = self.forward_features(x) 2024-08-21T00:16:31.9926119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 795, in forward_features 2024-08-21T00:16:31.9927076Z x = self.forward_tokens(x) 2024-08-21T00:16:31.9928073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 642, in forward_tokens 2024-08-21T00:16:31.9928924Z x = block(x) 2024-08-21T00:16:31.9929745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 135, in forward 2024-08-21T00:16:31.9930632Z x = x + self.drop_path1(self.attn(self.norm1(x))) 2024-08-21T00:16:31.9931632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 90, in forward 2024-08-21T00:16:31.9932790Z x = F.fold(x, output_size=(H, W), kernel_size=self.kernel_size, padding=self.padding, stride=self.stride) 2024-08-21T00:16:31.9933439Z 2024-08-21T00:17:12.2073057Z W0821 00:17:12.206000 37139 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:18:13.6060709Z pass 2024-08-21T00:18:13.6173241Z TIMING: entire_frame_compile:175.46803 code_gen:47.70394 inductor_compile:91.2463 backend_compile:145.93129 total_wall_time:211.79343 2024-08-21T00:18:13.6175000Z STATS: call_* op count: 1282 | FakeTensorMode.__torch_dispatch__:101369 | attempt fast:3996 | fast is_contiguous:3717 | FakeTensor.__torch_dispatch__:21028 | slow no contiguity match:279 | ProxyTorchDispatchMode.__torch_dispatch__:32632 2024-08-21T00:18:13.6176665Z Dynamo produced 3 graphs covering 1282 ops with 7 graph breaks (5 unique) 2024-08-21T00:18:29.1188211Z 2024-08-21T00:18:32.5652305Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:32.5652777Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:18:32.5653221Z cuda train xcit_large_24_p8_224 2024-08-21T00:18:36.8828703Z WARNING:common:fp64 golden ref were not generated for xcit_large_24_p8_224. Setting accuracy check to cosine 2024-08-21T00:22:53.1346023Z W0821 00:22:53.133000 37755 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:25:36.9543544Z pass_due_to_skip 2024-08-21T00:25:36.9972138Z TIMING: entire_frame_compile:393.16592 code_gen:102.35845 inductor_compile:200.58773 backend_compile:325.49991 total_wall_time:458.00387 2024-08-21T00:25:36.9974377Z STATS: call_* op count: 3042 | FakeTensorMode.__torch_dispatch__:238760 | attempt fast:10604 | fast is_contiguous:7432 | FakeTensor.__torch_dispatch__:57122 | slow no contiguity match:2258 | fast channels_last:914 | ProxyTorchDispatchMode.__torch_dispatch__:78438 2024-08-21T00:25:36.9978069Z Dynamo produced 3 graphs covering 3042 ops with 7 graph breaks (5 unique) 2024-08-21T00:25:55.0055577Z accuracy pass_rate=96.77% 2024-08-21T00:25:55.0058736Z calls_captured gmean=691.71x mean=888.258x 2024-08-21T00:25:55.0061328Z unique_graphs gmean=2.49x mean=2.548x 2024-08-21T00:25:55.0064820Z graph_breaks gmean=6.39x mean=6.419x 2024-08-21T00:25:55.0068339Z unique_graph_breaks gmean=4.96x mean=4.968x 2024-08-21T00:25:55.0072124Z autograd_captures gmean=0.00x mean=0.000x 2024-08-21T00:25:55.0075589Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-21T00:25:55.0079134Z cudagraph_skips gmean=0.00x mean=0.097x 2024-08-21T00:25:57.6220738Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_timm_training.csv 2024-08-21T00:25:57.8850956Z mixnet_l PASS 2024-08-21T00:25:57.8855601Z mnasnet_100 PASS 2024-08-21T00:25:57.8859156Z mobilenetv2_100 PASS 2024-08-21T00:25:57.8863687Z mobilenetv3_large_100 PASS 2024-08-21T00:25:57.8868357Z mobilevit_s PASS 2024-08-21T00:25:57.8873008Z nfnet_l0 PASS 2024-08-21T00:25:57.8877949Z pit_b_224 PASS 2024-08-21T00:25:57.8882142Z pnasnet5large PASS 2024-08-21T00:25:57.8886733Z poolformer_m36 PASS 2024-08-21T00:25:57.8891333Z regnety_002 PASS 2024-08-21T00:25:57.8895955Z repvgg_a2 PASS 2024-08-21T00:25:57.8900305Z res2net101_26w_4s PASS 2024-08-21T00:25:57.8904772Z res2net50_14w_8s PASS 2024-08-21T00:25:57.8909459Z res2next50 PASS 2024-08-21T00:25:57.8913979Z resmlp_12_224 PASS 2024-08-21T00:25:57.8918844Z resnest101e PASS 2024-08-21T00:25:57.8923731Z rexnet_100 PASS 2024-08-21T00:25:57.8928215Z sebotnet33ts_256 PASS 2024-08-21T00:25:57.8933221Z selecsls42b PASS 2024-08-21T00:25:57.8937755Z spnasnet_100 PASS 2024-08-21T00:25:57.8942225Z swin_base_patch4_window7_224 PASS 2024-08-21T00:25:57.8946828Z swsl_resnext101_32x16d PASS 2024-08-21T00:25:57.8951688Z tf_efficientnet_b0 PASS 2024-08-21T00:25:57.8956044Z tf_mixnet_l PASS 2024-08-21T00:25:57.8960723Z tinynet_a PASS 2024-08-21T00:25:57.8965200Z tnt_s_patch16_224 PASS 2024-08-21T00:25:57.8969734Z twins_pcpvt_base PASS 2024-08-21T00:25:57.8974197Z visformer_small PASS 2024-08-21T00:25:57.8978797Z vit_base_patch16_224 PASS 2024-08-21T00:25:57.8983286Z volo_d1_224 PASS 2024-08-21T00:25:57.8987843Z xcit_large_24_p8_224 XFAIL 2024-08-21T00:25:57.9408832Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_timm_training.csv 2024-08-21T00:25:58.2100263Z mixnet_l PASS 2024-08-21T00:25:58.2104515Z mnasnet_100 PASS 2024-08-21T00:25:58.2110367Z mobilenetv2_100 PASS 2024-08-21T00:25:58.2115726Z mobilenetv3_large_100 PASS 2024-08-21T00:25:58.2121782Z mobilevit_s PASS 2024-08-21T00:25:58.2127128Z nfnet_l0 PASS 2024-08-21T00:25:58.2132337Z pit_b_224 PASS 2024-08-21T00:25:58.2137416Z pnasnet5large PASS 2024-08-21T00:25:58.2142728Z poolformer_m36 PASS 2024-08-21T00:25:58.2147959Z regnety_002 PASS 2024-08-21T00:25:58.2153209Z repvgg_a2 PASS 2024-08-21T00:25:58.2158266Z res2net101_26w_4s PASS 2024-08-21T00:25:58.2163309Z res2net50_14w_8s PASS 2024-08-21T00:25:58.2168495Z res2next50 PASS 2024-08-21T00:25:58.2173594Z resmlp_12_224 PASS 2024-08-21T00:25:58.2178745Z resnest101e PASS 2024-08-21T00:25:58.2183762Z rexnet_100 PASS 2024-08-21T00:25:58.2189099Z sebotnet33ts_256 PASS 2024-08-21T00:25:58.2194371Z selecsls42b PASS 2024-08-21T00:25:58.2199484Z spnasnet_100 PASS 2024-08-21T00:25:58.2204519Z swin_base_patch4_window7_224 PASS 2024-08-21T00:25:58.2209673Z swsl_resnext101_32x16d PASS 2024-08-21T00:25:58.2214765Z tf_efficientnet_b0 PASS 2024-08-21T00:25:58.2220810Z tf_mixnet_l PASS 2024-08-21T00:25:58.2225849Z tinynet_a PASS 2024-08-21T00:25:58.2231251Z tnt_s_patch16_224 PASS 2024-08-21T00:25:58.2236400Z twins_pcpvt_base PASS 2024-08-21T00:25:58.2241442Z visformer_small PASS 2024-08-21T00:25:58.2246849Z vit_base_patch16_224 PASS 2024-08-21T00:25:58.2251861Z volo_d1_224 PASS 2024-08-21T00:25:58.2256959Z xcit_large_24_p8_224 PASS 2024-08-21T00:25:58.2685367Z + cleanup_workspace 2024-08-21T00:25:58.2686241Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-21T00:25:58.2687345Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-21T00:25:58.2688307Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-21T00:25:58.2688950Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-21T00:25:58.2689752Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-21T00:25:58.2694577Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-21T00:25:58.2695408Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-21T00:25:59.0201041Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-21T00:25:59.0201576Z cat test/**/*_toprint.log || true 2024-08-21T00:25:59.0214915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:25:59.0215425Z env: 2024-08-21T00:25:59.0215706Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.0216161Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.0217244Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.0217903Z ##[endgroup] 2024-08-21T00:25:59.0313345Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-21T00:25:59.0345656Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:25:59.0346127Z kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:25:59.0354410Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:25:59.0354914Z env: 2024-08-21T00:25:59.0355190Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.0355639Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.0356379Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.0357037Z MONITOR_SCRIPT_PID: 52566 2024-08-21T00:25:59.0357402Z ##[endgroup] 2024-08-21T00:25:59.0550434Z Prepare all required actions 2024-08-21T00:25:59.0550926Z Getting action download info 2024-08-21T00:25:59.2630671Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-21T00:25:59.4461436Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-21T00:25:59.4461906Z with: 2024-08-21T00:25:59.4462492Z file-suffix: test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192 2024-08-21T00:25:59.4463198Z s3-bucket: gha-artifacts 2024-08-21T00:25:59.4463536Z env: 2024-08-21T00:25:59.4463810Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.4464264Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.4464999Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.4465640Z ##[endgroup] 2024-08-21T00:25:59.4492204Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-21T00:25:59.4492871Z # Remove any previous test jsons if they exist 2024-08-21T00:25:59.4493397Z rm -f test-jsons-*.zip 2024-08-21T00:25:59.4493927Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-21T00:25:59.4502941Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:25:59.4503444Z env: 2024-08-21T00:25:59.4503725Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.4504185Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.4504924Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.4505862Z FILE_SUFFIX: test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192 2024-08-21T00:25:59.4506547Z ##[endgroup] 2024-08-21T00:25:59.4852429Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-21T00:25:59.4881773Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-21T00:25:59.4882449Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-21T00:25:59.4889008Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-21T00:25:59.4894731Z adding: test/slow_tests.json (deflated 82%) 2024-08-21T00:25:59.4927429Z ##[group]Run # Remove any previous test reports if they exist 2024-08-21T00:25:59.4928066Z # Remove any previous test reports if they exist 2024-08-21T00:25:59.4928624Z rm -f test-reports-*.zip 2024-08-21T00:25:59.4929204Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-21T00:25:59.4938202Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:25:59.4938748Z env: 2024-08-21T00:25:59.4939021Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.4939467Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.4940192Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.4941119Z FILE_SUFFIX: test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192 2024-08-21T00:25:59.4941794Z ##[endgroup] 2024-08-21T00:25:59.5203304Z adding: test/test-reports/inference_timm_models.csv (deflated 69%) 2024-08-21T00:25:59.5204770Z adding: test/test-reports/training_timm_models.csv (deflated 66%) 2024-08-21T00:25:59.5208138Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-08-21T00:25:59.5209085Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 74%) 2024-08-21T00:25:59.5241189Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-21T00:25:59.5241822Z # Remove any previous usage logs if they exist 2024-08-21T00:25:59.5242321Z rm -f logs-*.zip 2024-08-21T00:25:59.5242990Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-21T00:25:59.5243777Z # so check to see if the file exists first 2024-08-21T00:25:59.5244281Z if [ -f 'usage_log.txt' ]; then 2024-08-21T00:25:59.5245009Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-21T00:25:59.5245497Z fi 2024-08-21T00:25:59.5245860Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-21T00:25:59.5246433Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-21T00:25:59.5246922Z fi 2024-08-21T00:25:59.5255497Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:25:59.5255994Z env: 2024-08-21T00:25:59.5256273Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.5256716Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.5257450Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.5258395Z FILE_SUFFIX: test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192 2024-08-21T00:25:59.5259105Z ##[endgroup] 2024-08-21T00:25:59.5371395Z adding: usage_log.txt (deflated 92%) 2024-08-21T00:25:59.5461586Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-21T00:25:59.5462315Z # Remove any previous debugging artifacts if they exist 2024-08-21T00:25:59.5462870Z rm -f debug-*.zip 2024-08-21T00:25:59.5463258Z if [ -d 'test/debug' ]; then 2024-08-21T00:25:59.5463758Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-21T00:25:59.5464223Z fi 2024-08-21T00:25:59.5472414Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:25:59.5472955Z env: 2024-08-21T00:25:59.5473245Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.5473718Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.5474526Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.5475577Z FILE_SUFFIX: test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192 2024-08-21T00:25:59.5476334Z ##[endgroup] 2024-08-21T00:25:59.5572442Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:25:59.5573068Z with: 2024-08-21T00:25:59.5573367Z s3-bucket: gha-artifacts 2024-08-21T00:25:59.5573802Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:25:59.5574275Z retention-days: 14 2024-08-21T00:25:59.5574611Z if-no-files-found: warn 2024-08-21T00:25:59.5574974Z path: test-jsons-*.zip 2024-08-21T00:25:59.5575315Z name: artifact 2024-08-21T00:25:59.5575615Z region: us-east-1 2024-08-21T00:25:59.5575918Z env: 2024-08-21T00:25:59.5576205Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:25:59.5576659Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:25:59.5577393Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:25:59.5578035Z ##[endgroup] 2024-08-21T00:25:59.9075843Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:25:59.9076445Z With the provided path, there will be 1 file uploaded 2024-08-21T00:25:59.9077066Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:26:00.0424292Z Starting upload of test-jsons-test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192.zip 2024-08-21T00:26:00.1573224Z Finished upload of test-jsons-test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192.zip 2024-08-21T00:26:00.1851599Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:26:00.1852051Z with: 2024-08-21T00:26:00.1852343Z s3-bucket: gha-artifacts 2024-08-21T00:26:00.1852782Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:26:00.1853258Z retention-days: 14 2024-08-21T00:26:00.1853597Z if-no-files-found: error 2024-08-21T00:26:00.1853973Z path: test-reports-*.zip 2024-08-21T00:26:00.1854329Z name: artifact 2024-08-21T00:26:00.1854640Z region: us-east-1 2024-08-21T00:26:00.1854955Z env: 2024-08-21T00:26:00.1855247Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:26:00.1855709Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:26:00.1856448Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:00.1857316Z ##[endgroup] 2024-08-21T00:26:00.5365223Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:26:00.5366023Z With the provided path, there will be 1 file uploaded 2024-08-21T00:26:00.5366674Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:26:00.5423542Z Starting upload of test-reports-test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192.zip 2024-08-21T00:26:00.6621281Z Finished upload of test-reports-test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192.zip 2024-08-21T00:26:00.6902087Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:26:00.6902535Z with: 2024-08-21T00:26:00.6902834Z s3-bucket: gha-artifacts 2024-08-21T00:26:00.6903288Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:26:00.6903779Z retention-days: 14 2024-08-21T00:26:00.6904121Z if-no-files-found: ignore 2024-08-21T00:26:00.6904489Z path: logs-*.zip 2024-08-21T00:26:00.6904838Z name: artifact 2024-08-21T00:26:00.6905150Z region: us-east-1 2024-08-21T00:26:00.6905467Z env: 2024-08-21T00:26:00.6905755Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:26:00.6906223Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:26:00.6906961Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:00.6907611Z ##[endgroup] 2024-08-21T00:26:01.0465906Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:26:01.0467332Z With the provided path, there will be 1 file uploaded 2024-08-21T00:26:01.0468357Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:26:01.0522414Z Starting upload of logs-test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192.zip 2024-08-21T00:26:01.1712375Z Finished upload of logs-test-dynamic_inductor_timm-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529192.zip 2024-08-21T00:26:01.1993481Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:26:01.1994118Z with: 2024-08-21T00:26:01.1994417Z s3-bucket: gha-artifacts 2024-08-21T00:26:01.1994859Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:26:01.1995335Z retention-days: 14 2024-08-21T00:26:01.1995674Z if-no-files-found: ignore 2024-08-21T00:26:01.1996043Z path: debug-*.zip 2024-08-21T00:26:01.1996359Z name: artifact 2024-08-21T00:26:01.1996657Z region: us-east-1 2024-08-21T00:26:01.1996957Z env: 2024-08-21T00:26:01.1997231Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:26:01.1997679Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:26:01.1998407Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:01.1999042Z ##[endgroup] 2024-08-21T00:26:01.5413859Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-21T00:26:01.5694309Z ##[group]Run # shellcheck disable=SC2156 2024-08-21T00:26:01.5694813Z # shellcheck disable=SC2156 2024-08-21T00:26:01.5695630Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-21T00:26:01.5704772Z shell: /usr/bin/bash -e {0} 2024-08-21T00:26:01.5705132Z env: 2024-08-21T00:26:01.5705414Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:26:01.5705875Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:26:01.5706607Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:01.5707248Z ##[endgroup] 2024-08-21T00:26:01.8313956Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-21T00:26:01.8314512Z with: 2024-08-21T00:26:01.8314763Z env: 2024-08-21T00:26:01.8315034Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:26:01.8315476Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:26:01.8316210Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:01.8316840Z ##[endgroup] 2024-08-21T00:26:01.8336576Z ##[group]Run set -eou pipefail 2024-08-21T00:26:01.8336990Z set -eou pipefail 2024-08-21T00:26:01.8337351Z  2024-08-21T00:26:01.8337866Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-21T00:26:01.8338505Z for _ in $(seq 1440); do 2024-08-21T00:26:01.8338988Z  # Break if no ssh session exists anymore 2024-08-21T00:26:01.8339493Z  if [ "$(who)" = "" ]; then 2024-08-21T00:26:01.8339908Z  break 2024-08-21T00:26:01.8340259Z  fi 2024-08-21T00:26:01.8340567Z  echo "." 2024-08-21T00:26:01.8340907Z  sleep 5 2024-08-21T00:26:01.8341232Z done 2024-08-21T00:26:01.8350039Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:26:01.8350543Z env: 2024-08-21T00:26:01.8350832Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:26:01.8351289Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:26:01.8352021Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:01.8352670Z ##[endgroup] 2024-08-21T00:26:01.8382124Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-21T00:26:01.8493096Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:26:01.8493863Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:26:01.8494454Z # shellcheck disable=SC2046 2024-08-21T00:26:01.8494932Z docker stop $(docker ps -q) || true 2024-08-21T00:26:01.8495405Z # Prune all of the docker images 2024-08-21T00:26:01.8495854Z docker system prune -af 2024-08-21T00:26:01.8504697Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:26:01.8505203Z env: 2024-08-21T00:26:01.8505487Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:26:01.8505943Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:26:01.8506670Z DOCKER_CONTAINER_ID: ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:01.8507485Z ##[endgroup] 2024-08-21T00:26:02.3680141Z ef5eb1aef7f0 2024-08-21T00:26:03.8194838Z Deleted Containers: 2024-08-21T00:26:03.8195531Z ef5eb1aef7f0d973f4122c77ef75931834196a4d4382e46f35127ce6775f959e 2024-08-21T00:26:03.8196108Z 2024-08-21T00:26:14.8994389Z Deleted Images: 2024-08-21T00:26:14.8996363Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T00:26:14.8998735Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-21T00:26:14.9000285Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-21T00:26:14.9001399Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-21T00:26:14.9002484Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-21T00:26:14.9003538Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-21T00:26:14.9004527Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-21T00:26:14.9005316Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-21T00:26:14.9006124Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-21T00:26:14.9007275Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-21T00:26:14.9008105Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-21T00:26:14.9008979Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-21T00:26:14.9009800Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-21T00:26:14.9010607Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-21T00:26:14.9011483Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-21T00:26:14.9012304Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-21T00:26:14.9013114Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-21T00:26:14.9013931Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-21T00:26:14.9014763Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-21T00:26:14.9015594Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-21T00:26:14.9016409Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-21T00:26:14.9017659Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-21T00:26:14.9018486Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-21T00:26:14.9019324Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-21T00:26:14.9020143Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-21T00:26:14.9021001Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-21T00:26:14.9021824Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-21T00:26:14.9022663Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-21T00:26:14.9023497Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-21T00:26:14.9024324Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-21T00:26:14.9025146Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-21T00:26:14.9025968Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-21T00:26:14.9026995Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-21T00:26:14.9027822Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-21T00:26:14.9028892Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-21T00:26:14.9029933Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-21T00:26:14.9030971Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-21T00:26:14.9032010Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-21T00:26:14.9033058Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-21T00:26:14.9034129Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-21T00:26:14.9035204Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-21T00:26:14.9036283Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-21T00:26:14.9037376Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-21T00:26:14.9038479Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-21T00:26:14.9039539Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-21T00:26:14.9040567Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-21T00:26:14.9041811Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-21T00:26:14.9042885Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-21T00:26:14.9043966Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-21T00:26:14.9044990Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-21T00:26:14.9046038Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-21T00:26:14.9047132Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-21T00:26:14.9048220Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-21T00:26:14.9049298Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-21T00:26:14.9050372Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-21T00:26:14.9051470Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-21T00:26:14.9052595Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-21T00:26:14.9053619Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-21T00:26:14.9054439Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-21T00:26:14.9055243Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-21T00:26:14.9056055Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-21T00:26:14.9056953Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-21T00:26:14.9058069Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-21T00:26:14.9059107Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-21T00:26:14.9059939Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-21T00:26:14.9060749Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-21T00:26:14.9061565Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-21T00:26:14.9062394Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-21T00:26:14.9063211Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-21T00:26:14.9064035Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-21T00:26:14.9064994Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-21T00:26:14.9065809Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-21T00:26:14.9066605Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-21T00:26:14.9067415Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-21T00:26:14.9068334Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-21T00:26:14.9069188Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-21T00:26:14.9070019Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-21T00:26:14.9070853Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-21T00:26:14.9071689Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-21T00:26:14.9072528Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-21T00:26:14.9073339Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-21T00:26:14.9073834Z 2024-08-21T00:26:14.9073986Z Total reclaimed space: 44.87GB 2024-08-21T00:26:14.9145255Z Post job cleanup. 2024-08-21T00:26:14.9195637Z Post job cleanup. 2024-08-21T00:26:15.0073357Z [command]/usr/bin/git version 2024-08-21T00:26:15.0140355Z git version 2.40.1 2024-08-21T00:26:15.0176918Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/53c666e9-72d7-4595-82ed-a80db66a42a9' before making global git config changes 2024-08-21T00:26:15.0178132Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T00:26:15.0181947Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-21T00:26:15.0231411Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T00:26:15.0275490Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T00:26:15.0659058Z Entering 'android/libs/fbjni' 2024-08-21T00:26:15.0728907Z Entering 'third_party/FP16' 2024-08-21T00:26:15.0797185Z Entering 'third_party/FXdiv' 2024-08-21T00:26:15.0867818Z Entering 'third_party/NNPACK' 2024-08-21T00:26:15.0937803Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:26:15.1005532Z Entering 'third_party/XNNPACK' 2024-08-21T00:26:15.1091217Z Entering 'third_party/benchmark' 2024-08-21T00:26:15.1160434Z Entering 'third_party/cpp-httplib' 2024-08-21T00:26:15.1228380Z Entering 'third_party/cpuinfo' 2024-08-21T00:26:15.1296443Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:26:15.1366689Z Entering 'third_party/cutlass' 2024-08-21T00:26:15.1441837Z Entering 'third_party/eigen' 2024-08-21T00:26:15.1511603Z Entering 'third_party/fbgemm' 2024-08-21T00:26:15.1578776Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:26:15.1644174Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:26:15.1710798Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:26:15.1785542Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:26:15.1854704Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:26:15.1924873Z Entering 'third_party/flatbuffers' 2024-08-21T00:26:15.1996616Z Entering 'third_party/fmt' 2024-08-21T00:26:15.2066890Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:26:15.2134516Z Entering 'third_party/gloo' 2024-08-21T00:26:15.2204727Z Entering 'third_party/googletest' 2024-08-21T00:26:15.2274389Z Entering 'third_party/ideep' 2024-08-21T00:26:15.2343283Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:26:15.2417613Z Entering 'third_party/ittapi' 2024-08-21T00:26:15.2486873Z Entering 'third_party/kineto' 2024-08-21T00:26:15.2554179Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:26:15.2619746Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:26:15.2687645Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:26:15.2755966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:26:15.2824634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:26:15.2891334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:26:15.2962955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:26:15.3030143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:26:15.3098873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:26:15.3168254Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:26:15.3239043Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:26:15.3305393Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:26:15.3377852Z Entering 'third_party/mimalloc' 2024-08-21T00:26:15.3446826Z Entering 'third_party/nccl/nccl' 2024-08-21T00:26:15.3515836Z Entering 'third_party/nlohmann' 2024-08-21T00:26:15.3584657Z Entering 'third_party/onnx' 2024-08-21T00:26:15.3670005Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:26:15.3737186Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:26:15.3811192Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:26:15.3881855Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:26:15.3949209Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:26:15.4015917Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:26:15.4083967Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:26:15.4152803Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:26:15.4218462Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:26:15.4284656Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:26:15.4349563Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:26:15.4418296Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:26:15.4488480Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:26:15.4577192Z Entering 'third_party/pocketfft' 2024-08-21T00:26:15.4647271Z Entering 'third_party/protobuf' 2024-08-21T00:26:15.4718223Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:26:15.4787010Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:26:15.4856536Z Entering 'third_party/psimd' 2024-08-21T00:26:15.4924330Z Entering 'third_party/pthreadpool' 2024-08-21T00:26:15.4992314Z Entering 'third_party/pybind11' 2024-08-21T00:26:15.5060973Z Entering 'third_party/python-peachpy' 2024-08-21T00:26:15.5128636Z Entering 'third_party/sleef' 2024-08-21T00:26:15.5196891Z Entering 'third_party/tensorpipe' 2024-08-21T00:26:15.5263707Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:26:15.5337156Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:26:15.5402040Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:26:15.5468490Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:26:15.5533589Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:26:15.5627905Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T00:26:15.5666405Z http.https://github.com/.extraheader 2024-08-21T00:26:15.5676259Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-21T00:26:15.5722670Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T00:26:15.6087369Z Entering 'android/libs/fbjni' 2024-08-21T00:26:15.6131942Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6175688Z Entering 'third_party/FP16' 2024-08-21T00:26:15.6220093Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6262900Z Entering 'third_party/FXdiv' 2024-08-21T00:26:15.6308284Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6353319Z Entering 'third_party/NNPACK' 2024-08-21T00:26:15.6401405Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6444344Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:26:15.6488411Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6535702Z Entering 'third_party/XNNPACK' 2024-08-21T00:26:15.6578808Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6638373Z Entering 'third_party/benchmark' 2024-08-21T00:26:15.6681401Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6725807Z Entering 'third_party/cpp-httplib' 2024-08-21T00:26:15.6770376Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6814237Z Entering 'third_party/cpuinfo' 2024-08-21T00:26:15.6858332Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6904129Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:26:15.6947529Z http.https://github.com/.extraheader 2024-08-21T00:26:15.6992941Z Entering 'third_party/cutlass' 2024-08-21T00:26:15.7037102Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7088100Z Entering 'third_party/eigen' 2024-08-21T00:26:15.7132185Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7179033Z Entering 'third_party/fbgemm' 2024-08-21T00:26:15.7227941Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7273058Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:26:15.7315520Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7359236Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:26:15.7402243Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7446897Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:26:15.7488932Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7538688Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:26:15.7581232Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7625008Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:26:15.7667289Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7714779Z Entering 'third_party/flatbuffers' 2024-08-21T00:26:15.7759864Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7807905Z Entering 'third_party/fmt' 2024-08-21T00:26:15.7854141Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7898551Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:26:15.7943469Z http.https://github.com/.extraheader 2024-08-21T00:26:15.7987607Z Entering 'third_party/gloo' 2024-08-21T00:26:15.8032238Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8075726Z Entering 'third_party/googletest' 2024-08-21T00:26:15.8120266Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8166140Z Entering 'third_party/ideep' 2024-08-21T00:26:15.8210714Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8253618Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:26:15.8299277Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8350785Z Entering 'third_party/ittapi' 2024-08-21T00:26:15.8395040Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8439741Z Entering 'third_party/kineto' 2024-08-21T00:26:15.8484251Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8528324Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:26:15.8573576Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8616595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:26:15.8659334Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8704492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:26:15.8746765Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8790598Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:26:15.8833928Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8877856Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:26:15.8921263Z http.https://github.com/.extraheader 2024-08-21T00:26:15.8964787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:26:15.9007765Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9058102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:26:15.9100858Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9146230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:26:15.9189124Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9234518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:26:15.9277522Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9323751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:26:15.9371155Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9417899Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:26:15.9463454Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9506434Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:26:15.9549407Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9595441Z Entering 'third_party/mimalloc' 2024-08-21T00:26:15.9640756Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9686500Z Entering 'third_party/nccl/nccl' 2024-08-21T00:26:15.9733843Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9778262Z Entering 'third_party/nlohmann' 2024-08-21T00:26:15.9822084Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9866382Z Entering 'third_party/onnx' 2024-08-21T00:26:15.9909719Z http.https://github.com/.extraheader 2024-08-21T00:26:15.9967229Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:26:16.0010074Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0056428Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:26:16.0098376Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0148893Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:26:16.0192899Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0237835Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:26:16.0279717Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0324013Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:26:16.0366729Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0409427Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:26:16.0452130Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0497882Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:26:16.0540505Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0588905Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:26:16.0632962Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0675308Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:26:16.0717340Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0759918Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:26:16.0801961Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0843738Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:26:16.0886524Z http.https://github.com/.extraheader 2024-08-21T00:26:16.0933300Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:26:16.0975542Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1023161Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:26:16.1065068Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1129396Z Entering 'third_party/pocketfft' 2024-08-21T00:26:16.1172509Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1215378Z Entering 'third_party/protobuf' 2024-08-21T00:26:16.1258907Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1303767Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:26:16.1346721Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1388989Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:26:16.1432579Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1480083Z Entering 'third_party/psimd' 2024-08-21T00:26:16.1524801Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1568295Z Entering 'third_party/pthreadpool' 2024-08-21T00:26:16.1611685Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1655974Z Entering 'third_party/pybind11' 2024-08-21T00:26:16.1699850Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1744652Z Entering 'third_party/python-peachpy' 2024-08-21T00:26:16.1789342Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1833752Z Entering 'third_party/sleef' 2024-08-21T00:26:16.1878148Z http.https://github.com/.extraheader 2024-08-21T00:26:16.1923550Z Entering 'third_party/tensorpipe' 2024-08-21T00:26:16.1967481Z http.https://github.com/.extraheader 2024-08-21T00:26:16.2010614Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:26:16.2053402Z http.https://github.com/.extraheader 2024-08-21T00:26:16.2096798Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:26:16.2137874Z http.https://github.com/.extraheader 2024-08-21T00:26:16.2179678Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:26:16.2222311Z http.https://github.com/.extraheader 2024-08-21T00:26:16.2265016Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:26:16.2310443Z http.https://github.com/.extraheader 2024-08-21T00:26:16.2352257Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:26:16.2395348Z http.https://github.com/.extraheader 2024-08-21T00:26:16.2553726Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-21T00:26:16.2576359Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-21T00:26:16.2584404Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:26:16.2584925Z ##[endgroup] 2024-08-21T00:26:23.1138929Z Cleaning up orphan processes