2024-08-20T22:09:56.1045639Z Current runner version: '2.319.1' 2024-08-20T22:09:56.1051933Z Runner name: 'i-0035567a13af53524' 2024-08-20T22:09:56.1052863Z Runner group name: 'Default' 2024-08-20T22:09:56.1053733Z Machine name: 'ip-10-0-45-195' 2024-08-20T22:09:56.1070117Z Testing runner upgrade compatibility 2024-08-20T22:09:56.2483365Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:09:56.2485512Z Actions: read 2024-08-20T22:09:56.2486106Z Attestations: read 2024-08-20T22:09:56.2486659Z Checks: read 2024-08-20T22:09:56.2487157Z Contents: read 2024-08-20T22:09:56.2487747Z Deployments: read 2024-08-20T22:09:56.2488235Z Discussions: read 2024-08-20T22:09:56.2488746Z Issues: read 2024-08-20T22:09:56.2489403Z Metadata: read 2024-08-20T22:09:56.2489876Z Packages: read 2024-08-20T22:09:56.2490387Z Pages: read 2024-08-20T22:09:56.2490947Z PullRequests: read 2024-08-20T22:09:56.2491507Z RepositoryProjects: read 2024-08-20T22:09:56.2492070Z SecurityEvents: read 2024-08-20T22:09:56.2492690Z Statuses: read 2024-08-20T22:09:56.2493370Z ##[endgroup] 2024-08-20T22:09:56.2496553Z Secret source: Actions 2024-08-20T22:09:56.2497407Z Prepare workflow directory 2024-08-20T22:09:56.7954470Z Prepare all required actions 2024-08-20T22:09:56.8138534Z Getting action download info 2024-08-20T22:09:57.0051852Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:09:57.5008631Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:10:01.5337816Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:10:01.7406449Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:10:01.9794702Z Getting action download info 2024-08-20T22:10:02.0708470Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:10:02.2502287Z Getting action download info 2024-08-20T22:10:02.3458024Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:10:02.4915451Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:10:02.4917649Z ##[group] Inputs 2024-08-20T22:10:02.4918115Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:10:02.4927622Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:10:02.4938298Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:02.4939505Z sync-tag: 2024-08-20T22:10:02.4940397Z timeout-minutes: 240 2024-08-20T22:10:02.4940739Z use-gha: 2024-08-20T22:10:02.4941039Z dashboard-tag: 2024-08-20T22:10:02.4941373Z s3-bucket: gha-artifacts 2024-08-20T22:10:02.4941731Z aws-role-to-assume: 2024-08-20T22:10:02.4942071Z ##[endgroup] 2024-08-20T22:10:02.4942995Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:10:02.5621674Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:02.5763690Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:10:02.5775787Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:02.5776415Z ##[endgroup] 2024-08-20T22:10:04.5212044Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:10:04.5212606Z Instance Type: g5.4xlarge 2024-08-20T22:10:04.5213328Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:10:04.5213915Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:10.3497248Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:10.3497835Z with: 2024-08-20T22:10:10.3498529Z github-secret: *** 2024-08-20T22:10:10.3499465Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:10.3500468Z activate-with-label: false 2024-08-20T22:10:10.3500829Z label: with-ssh 2024-08-20T22:10:10.3501153Z remove-existing-keys: true 2024-08-20T22:10:10.3501523Z fail-silently: true 2024-08-20T22:10:10.3501829Z env: 2024-08-20T22:10:10.3502105Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.3502450Z ##[endgroup] 2024-08-20T22:10:10.4398222Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:10.4401082Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:10.7334163Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:10.8060630Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:10.8061249Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:10.8800568Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:10.8814814Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:10.8840359Z Login using: ssh ec2-user@ec2-44-210-124-102.compute-1.amazonaws.com 2024-08-20T22:10:10.8841169Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:10.8841988Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:10.8974897Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:10.8975464Z with: 2024-08-20T22:10:10.8975740Z submodules: recursive 2024-08-20T22:10:10.8976068Z fetch-depth: 0 2024-08-20T22:10:10.8976354Z env: 2024-08-20T22:10:10.8976623Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.8976951Z ##[endgroup] 2024-08-20T22:10:10.9170353Z ##[group]Run retry () { 2024-08-20T22:10:10.9170752Z retry () { 2024-08-20T22:10:10.9171514Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:10.9172076Z } 2024-08-20T22:10:10.9172381Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.9172806Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:10.9173284Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.9173743Z else 2024-08-20T22:10:10.9174081Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.9174510Z fi 2024-08-20T22:10:10.9174852Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.9185743Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:10.9186256Z env: 2024-08-20T22:10:10.9186536Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.9186956Z NO_SUDO: 2024-08-20T22:10:10.9187231Z ##[endgroup] 2024-08-20T22:10:10.9217437Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:15.4429951Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:15.4430401Z with: 2024-08-20T22:10:15.4430732Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:15.4431180Z fetch-depth: 0 2024-08-20T22:10:15.4431495Z submodules: recursive 2024-08-20T22:10:15.4431831Z quiet-checkout: true 2024-08-20T22:10:15.4432183Z repository: pytorch/pytorch 2024-08-20T22:10:15.4432680Z token: *** 2024-08-20T22:10:15.4432971Z ssh-strict: true 2024-08-20T22:10:15.4433291Z persist-credentials: true 2024-08-20T22:10:15.4434031Z clean: true 2024-08-20T22:10:15.4434372Z sparse-checkout-cone-mode: true 2024-08-20T22:10:15.4434766Z lfs: false 2024-08-20T22:10:15.4435063Z set-safe-directory: true 2024-08-20T22:10:15.4435405Z env: 2024-08-20T22:10:15.4435679Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:15.4436010Z ##[endgroup] 2024-08-20T22:10:15.5418723Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:15.5420196Z ##[group]Getting Git version info 2024-08-20T22:10:15.5420926Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:15.5421798Z [command]/usr/bin/git version 2024-08-20T22:10:15.5422163Z git version 2.40.1 2024-08-20T22:10:15.5425395Z ##[endgroup] 2024-08-20T22:10:15.5442202Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/944b4711-f31b-4b40-8d0c-6b81228699a1' before making global git config changes 2024-08-20T22:10:15.5443466Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:15.5445213Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:15.5497754Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:15.5501808Z ##[group]Initializing the repository 2024-08-20T22:10:15.5504502Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:15.5549955Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:15.5551072Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:15.5551910Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:15.5552480Z hint: 2024-08-20T22:10:15.5552942Z hint: git config --global init.defaultBranch 2024-08-20T22:10:15.5553418Z hint: 2024-08-20T22:10:15.5553943Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:15.5554791Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:15.5555703Z hint: 2024-08-20T22:10:15.5556048Z hint: git branch -m 2024-08-20T22:10:15.5556802Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:15.5563146Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:15.5605954Z ##[endgroup] 2024-08-20T22:10:15.5606553Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:15.5608714Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:15.5650725Z ##[endgroup] 2024-08-20T22:10:15.5651255Z ##[group]Setting up auth 2024-08-20T22:10:15.5656491Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:15.5699102Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:15.6053452Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:15.6094669Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:15.6448961Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:15.6509705Z ##[endgroup] 2024-08-20T22:10:15.6510292Z ##[group]Fetching the repository 2024-08-20T22:10:15.6515748Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:18.2403350Z remote: Enumerating objects: 1008452 2024-08-20T22:10:18.2403913Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:18.2404700Z remote: Counting objects: 0% (1/1777) 2024-08-20T22:10:18.2405230Z remote: Counting objects: 1% (18/1777) 2024-08-20T22:10:18.2405766Z remote: Counting objects: 2% (36/1777) 2024-08-20T22:10:18.2406391Z remote: Counting objects: 3% (54/1777) 2024-08-20T22:10:18.2406893Z remote: Counting objects: 4% (72/1777) 2024-08-20T22:10:18.2407398Z remote: Counting objects: 5% (89/1777) 2024-08-20T22:10:18.2407896Z remote: Counting objects: 6% (107/1777) 2024-08-20T22:10:18.2408418Z remote: Counting objects: 7% (125/1777) 2024-08-20T22:10:18.2408948Z remote: Counting objects: 8% (143/1777) 2024-08-20T22:10:18.2409448Z remote: Counting objects: 9% (160/1777) 2024-08-20T22:10:18.2409941Z remote: Counting objects: 10% (178/1777) 2024-08-20T22:10:18.2410450Z remote: Counting objects: 11% (196/1777) 2024-08-20T22:10:18.2410974Z remote: Counting objects: 12% (214/1777) 2024-08-20T22:10:18.2411478Z remote: Counting objects: 13% (232/1777) 2024-08-20T22:10:18.2411997Z remote: Counting objects: 14% (249/1777) 2024-08-20T22:10:18.2412539Z remote: Counting objects: 15% (267/1777) 2024-08-20T22:10:18.2413119Z remote: Counting objects: 16% (285/1777) 2024-08-20T22:10:18.2413622Z remote: Counting objects: 17% (303/1777) 2024-08-20T22:10:18.2414119Z remote: Counting objects: 18% (320/1777) 2024-08-20T22:10:18.2414608Z remote: Counting objects: 19% (338/1777) 2024-08-20T22:10:18.2415140Z remote: Counting objects: 20% (356/1777) 2024-08-20T22:10:18.2415796Z remote: Counting objects: 21% (374/1777) 2024-08-20T22:10:18.2416425Z remote: Counting objects: 22% (391/1777) 2024-08-20T22:10:18.2416915Z remote: Counting objects: 23% (409/1777) 2024-08-20T22:10:18.2417417Z remote: Counting objects: 24% (427/1777) 2024-08-20T22:10:18.2417915Z remote: Counting objects: 25% (445/1777) 2024-08-20T22:10:18.2418428Z remote: Counting objects: 26% (463/1777) 2024-08-20T22:10:18.2418924Z remote: Counting objects: 27% (480/1777) 2024-08-20T22:10:18.2419842Z remote: Counting objects: 28% (498/1777) 2024-08-20T22:10:18.2420366Z remote: Counting objects: 29% (516/1777) 2024-08-20T22:10:18.2420858Z remote: Counting objects: 30% (534/1777) 2024-08-20T22:10:18.2421366Z remote: Counting objects: 31% (551/1777) 2024-08-20T22:10:18.2421957Z remote: Counting objects: 32% (569/1777) 2024-08-20T22:10:18.2422460Z remote: Counting objects: 33% (587/1777) 2024-08-20T22:10:18.2423142Z remote: Counting objects: 34% (605/1777) 2024-08-20T22:10:18.2423637Z remote: Counting objects: 35% (622/1777) 2024-08-20T22:10:18.2424154Z remote: Counting objects: 36% (640/1777) 2024-08-20T22:10:18.2424654Z remote: Counting objects: 37% (658/1777) 2024-08-20T22:10:18.2425207Z remote: Counting objects: 38% (676/1777) 2024-08-20T22:10:18.2425721Z remote: Counting objects: 39% (694/1777) 2024-08-20T22:10:18.2426220Z remote: Counting objects: 40% (711/1777) 2024-08-20T22:10:18.2426730Z remote: Counting objects: 41% (729/1777) 2024-08-20T22:10:18.2427346Z remote: Counting objects: 42% (747/1777) 2024-08-20T22:10:18.2427837Z remote: Counting objects: 43% (765/1777) 2024-08-20T22:10:18.2428339Z remote: Counting objects: 44% (782/1777) 2024-08-20T22:10:18.2428836Z remote: Counting objects: 45% (800/1777) 2024-08-20T22:10:18.2429334Z remote: Counting objects: 46% (818/1777) 2024-08-20T22:10:18.2429834Z remote: Counting objects: 47% (836/1777) 2024-08-20T22:10:18.2430334Z remote: Counting objects: 48% (853/1777) 2024-08-20T22:10:18.2430841Z remote: Counting objects: 49% (871/1777) 2024-08-20T22:10:18.2431371Z remote: Counting objects: 50% (889/1777) 2024-08-20T22:10:18.2431861Z remote: Counting objects: 51% (907/1777) 2024-08-20T22:10:18.2432361Z remote: Counting objects: 52% (925/1777) 2024-08-20T22:10:18.2432861Z remote: Counting objects: 53% (942/1777) 2024-08-20T22:10:18.2433671Z remote: Counting objects: 54% (960/1777) 2024-08-20T22:10:18.2434182Z remote: Counting objects: 55% (978/1777) 2024-08-20T22:10:18.2434682Z remote: Counting objects: 56% (996/1777) 2024-08-20T22:10:18.2435187Z remote: Counting objects: 57% (1013/1777) 2024-08-20T22:10:18.2435713Z remote: Counting objects: 58% (1031/1777) 2024-08-20T22:10:18.2436229Z remote: Counting objects: 59% (1049/1777) 2024-08-20T22:10:18.2436754Z remote: Counting objects: 60% (1067/1777) 2024-08-20T22:10:18.2437271Z remote: Counting objects: 61% (1084/1777) 2024-08-20T22:10:18.2437791Z remote: Counting objects: 62% (1102/1777) 2024-08-20T22:10:18.2438317Z remote: Counting objects: 63% (1120/1777) 2024-08-20T22:10:18.2438830Z remote: Counting objects: 64% (1138/1777) 2024-08-20T22:10:18.2439337Z remote: Counting objects: 65% (1156/1777) 2024-08-20T22:10:18.2439855Z remote: Counting objects: 66% (1173/1777) 2024-08-20T22:10:18.2440366Z remote: Counting objects: 67% (1191/1777) 2024-08-20T22:10:18.2440871Z remote: Counting objects: 68% (1209/1777) 2024-08-20T22:10:18.2441383Z remote: Counting objects: 69% (1227/1777) 2024-08-20T22:10:18.2441891Z remote: Counting objects: 70% (1244/1777) 2024-08-20T22:10:18.2442399Z remote: Counting objects: 71% (1262/1777) 2024-08-20T22:10:18.2442903Z remote: Counting objects: 72% (1280/1777) 2024-08-20T22:10:18.2443418Z remote: Counting objects: 73% (1298/1777) 2024-08-20T22:10:18.2443928Z remote: Counting objects: 74% (1315/1777) 2024-08-20T22:10:18.2444435Z remote: Counting objects: 75% (1333/1777) 2024-08-20T22:10:18.2444936Z remote: Counting objects: 76% (1351/1777) 2024-08-20T22:10:18.2445447Z remote: Counting objects: 77% (1369/1777) 2024-08-20T22:10:18.2445955Z remote: Counting objects: 78% (1387/1777) 2024-08-20T22:10:18.2446646Z remote: Counting objects: 79% (1404/1777) 2024-08-20T22:10:18.2447181Z remote: Counting objects: 80% (1422/1777) 2024-08-20T22:10:18.2447690Z remote: Counting objects: 81% (1440/1777) 2024-08-20T22:10:18.2448197Z remote: Counting objects: 82% (1458/1777) 2024-08-20T22:10:18.2448706Z remote: Counting objects: 83% (1475/1777) 2024-08-20T22:10:18.2449210Z remote: Counting objects: 84% (1493/1777) 2024-08-20T22:10:18.2449717Z remote: Counting objects: 85% (1511/1777) 2024-08-20T22:10:18.2450348Z remote: Counting objects: 86% (1529/1777) 2024-08-20T22:10:18.2450859Z remote: Counting objects: 87% (1546/1777) 2024-08-20T22:10:18.2451363Z remote: Counting objects: 88% (1564/1777) 2024-08-20T22:10:18.2451872Z remote: Counting objects: 89% (1582/1777) 2024-08-20T22:10:18.2452381Z remote: Counting objects: 90% (1600/1777) 2024-08-20T22:10:18.2452887Z remote: Counting objects: 91% (1618/1777) 2024-08-20T22:10:18.2453394Z remote: Counting objects: 92% (1635/1777) 2024-08-20T22:10:18.2453912Z remote: Counting objects: 93% (1653/1777) 2024-08-20T22:10:18.2454421Z remote: Counting objects: 94% (1671/1777) 2024-08-20T22:10:18.2454932Z remote: Counting objects: 95% (1689/1777) 2024-08-20T22:10:18.2455436Z remote: Counting objects: 96% (1706/1777) 2024-08-20T22:10:18.2455942Z remote: Counting objects: 97% (1724/1777) 2024-08-20T22:10:18.2456473Z remote: Counting objects: 98% (1742/1777) 2024-08-20T22:10:18.2456981Z remote: Counting objects: 99% (1760/1777) 2024-08-20T22:10:18.2457484Z remote: Counting objects: 100% (1777/1777) 2024-08-20T22:10:18.2458029Z remote: Counting objects: 100% (1777/1777), done. 2024-08-20T22:10:18.2629629Z remote: Compressing objects: 0% (1/846) 2024-08-20T22:10:18.2903951Z remote: Compressing objects: 1% (9/846) 2024-08-20T22:10:18.3213309Z remote: Compressing objects: 2% (17/846) 2024-08-20T22:10:18.3548280Z remote: Compressing objects: 3% (26/846) 2024-08-20T22:10:18.4240584Z remote: Compressing objects: 4% (34/846) 2024-08-20T22:10:18.4720837Z remote: Compressing objects: 5% (43/846) 2024-08-20T22:10:18.5221554Z remote: Compressing objects: 6% (51/846) 2024-08-20T22:10:18.5601944Z remote: Compressing objects: 7% (60/846) 2024-08-20T22:10:18.5991712Z remote: Compressing objects: 8% (68/846) 2024-08-20T22:10:18.6200831Z remote: Compressing objects: 9% (77/846) 2024-08-20T22:10:18.6457453Z remote: Compressing objects: 10% (85/846) 2024-08-20T22:10:18.6644036Z remote: Compressing objects: 11% (94/846) 2024-08-20T22:10:18.6830227Z remote: Compressing objects: 12% (102/846) 2024-08-20T22:10:18.6992010Z remote: Compressing objects: 13% (110/846) 2024-08-20T22:10:18.7103078Z remote: Compressing objects: 14% (119/846) 2024-08-20T22:10:18.7202811Z remote: Compressing objects: 15% (127/846) 2024-08-20T22:10:18.7276457Z remote: Compressing objects: 16% (136/846) 2024-08-20T22:10:18.7350871Z remote: Compressing objects: 17% (144/846) 2024-08-20T22:10:18.7387101Z remote: Compressing objects: 18% (153/846) 2024-08-20T22:10:18.7439942Z remote: Compressing objects: 19% (161/846) 2024-08-20T22:10:18.7467227Z remote: Compressing objects: 20% (170/846) 2024-08-20T22:10:18.7488214Z remote: Compressing objects: 21% (178/846) 2024-08-20T22:10:18.7491843Z remote: Compressing objects: 22% (187/846) 2024-08-20T22:10:18.7499851Z remote: Compressing objects: 23% (195/846) 2024-08-20T22:10:18.7506466Z remote: Compressing objects: 24% (204/846) 2024-08-20T22:10:18.7511411Z remote: Compressing objects: 25% (212/846) 2024-08-20T22:10:18.7522053Z remote: Compressing objects: 26% (220/846) 2024-08-20T22:10:18.7525295Z remote: Compressing objects: 27% (229/846) 2024-08-20T22:10:18.7544173Z remote: Compressing objects: 28% (237/846) 2024-08-20T22:10:18.7556220Z remote: Compressing objects: 29% (246/846) 2024-08-20T22:10:18.7567523Z remote: Compressing objects: 30% (254/846) 2024-08-20T22:10:18.7578856Z remote: Compressing objects: 31% (263/846) 2024-08-20T22:10:18.7585178Z remote: Compressing objects: 32% (271/846) 2024-08-20T22:10:18.7595152Z remote: Compressing objects: 33% (280/846) 2024-08-20T22:10:18.7600678Z remote: Compressing objects: 34% (288/846) 2024-08-20T22:10:18.7608138Z remote: Compressing objects: 35% (297/846) 2024-08-20T22:10:18.7618822Z remote: Compressing objects: 36% (305/846) 2024-08-20T22:10:18.7624436Z remote: Compressing objects: 37% (314/846) 2024-08-20T22:10:18.7630433Z remote: Compressing objects: 38% (322/846) 2024-08-20T22:10:18.7638289Z remote: Compressing objects: 39% (330/846) 2024-08-20T22:10:18.7645941Z remote: Compressing objects: 40% (339/846) 2024-08-20T22:10:18.7650430Z remote: Compressing objects: 41% (347/846) 2024-08-20T22:10:18.7659999Z remote: Compressing objects: 42% (356/846) 2024-08-20T22:10:18.7670521Z remote: Compressing objects: 43% (364/846) 2024-08-20T22:10:18.7677733Z remote: Compressing objects: 44% (373/846) 2024-08-20T22:10:18.7683351Z remote: Compressing objects: 45% (381/846) 2024-08-20T22:10:18.7687264Z remote: Compressing objects: 46% (390/846) 2024-08-20T22:10:18.7691201Z remote: Compressing objects: 47% (398/846) 2024-08-20T22:10:18.7694904Z remote: Compressing objects: 48% (407/846) 2024-08-20T22:10:18.7698824Z remote: Compressing objects: 49% (415/846) 2024-08-20T22:10:18.7702324Z remote: Compressing objects: 50% (423/846) 2024-08-20T22:10:18.7705983Z remote: Compressing objects: 51% (432/846) 2024-08-20T22:10:18.7709455Z remote: Compressing objects: 52% (440/846) 2024-08-20T22:10:18.7712174Z remote: Compressing objects: 53% (449/846) 2024-08-20T22:10:18.7714711Z remote: Compressing objects: 54% (457/846) 2024-08-20T22:10:18.7717386Z remote: Compressing objects: 55% (466/846) 2024-08-20T22:10:18.7720259Z remote: Compressing objects: 56% (474/846) 2024-08-20T22:10:18.7720803Z remote: Compressing objects: 57% (483/846) 2024-08-20T22:10:18.7722743Z remote: Compressing objects: 58% (491/846) 2024-08-20T22:10:18.7723288Z remote: Compressing objects: 59% (500/846) 2024-08-20T22:10:18.7725419Z remote: Compressing objects: 60% (508/846) 2024-08-20T22:10:18.7726003Z remote: Compressing objects: 61% (517/846) 2024-08-20T22:10:18.7726568Z remote: Compressing objects: 62% (525/846) 2024-08-20T22:10:18.7727105Z remote: Compressing objects: 63% (533/846) 2024-08-20T22:10:18.7730244Z remote: Compressing objects: 64% (542/846) 2024-08-20T22:10:18.7736993Z remote: Compressing objects: 65% (550/846) 2024-08-20T22:10:18.7742775Z remote: Compressing objects: 66% (559/846) 2024-08-20T22:10:18.7747465Z remote: Compressing objects: 67% (567/846) 2024-08-20T22:10:18.7751676Z remote: Compressing objects: 68% (576/846) 2024-08-20T22:10:18.7757050Z remote: Compressing objects: 69% (584/846) 2024-08-20T22:10:18.7762818Z remote: Compressing objects: 70% (593/846) 2024-08-20T22:10:18.7765458Z remote: Compressing objects: 71% (601/846) 2024-08-20T22:10:18.7768089Z remote: Compressing objects: 72% (610/846) 2024-08-20T22:10:18.7772652Z remote: Compressing objects: 73% (618/846) 2024-08-20T22:10:18.7776288Z remote: Compressing objects: 74% (627/846) 2024-08-20T22:10:18.7780606Z remote: Compressing objects: 75% (635/846) 2024-08-20T22:10:18.7784029Z remote: Compressing objects: 76% (643/846) 2024-08-20T22:10:18.7787799Z remote: Compressing objects: 77% (652/846) 2024-08-20T22:10:18.7790368Z remote: Compressing objects: 78% (660/846) 2024-08-20T22:10:18.7793170Z remote: Compressing objects: 79% (669/846) 2024-08-20T22:10:18.7795897Z remote: Compressing objects: 80% (677/846) 2024-08-20T22:10:18.7798411Z remote: Compressing objects: 81% (686/846) 2024-08-20T22:10:18.7801602Z remote: Compressing objects: 82% (694/846) 2024-08-20T22:10:18.7804164Z remote: Compressing objects: 83% (703/846) 2024-08-20T22:10:18.7806697Z remote: Compressing objects: 84% (711/846) 2024-08-20T22:10:18.7809380Z remote: Compressing objects: 85% (720/846) 2024-08-20T22:10:18.7811776Z remote: Compressing objects: 86% (728/846) 2024-08-20T22:10:18.7814414Z remote: Compressing objects: 87% (737/846) 2024-08-20T22:10:18.7816916Z remote: Compressing objects: 88% (745/846) 2024-08-20T22:10:18.7819577Z remote: Compressing objects: 89% (753/846) 2024-08-20T22:10:18.7822031Z remote: Compressing objects: 90% (762/846) 2024-08-20T22:10:18.7824683Z remote: Compressing objects: 91% (770/846) 2024-08-20T22:10:18.7827391Z remote: Compressing objects: 92% (779/846) 2024-08-20T22:10:18.7827931Z remote: Compressing objects: 93% (787/846) 2024-08-20T22:10:18.7829730Z remote: Compressing objects: 94% (796/846) 2024-08-20T22:10:18.7832406Z remote: Compressing objects: 95% (804/846) 2024-08-20T22:10:18.7832943Z remote: Compressing objects: 96% (813/846) 2024-08-20T22:10:18.7835030Z remote: Compressing objects: 97% (821/846) 2024-08-20T22:10:18.7837409Z remote: Compressing objects: 98% (830/846) 2024-08-20T22:10:18.7839982Z remote: Compressing objects: 99% (838/846) 2024-08-20T22:10:18.7840515Z remote: Compressing objects: 100% (846/846) 2024-08-20T22:10:18.7841084Z remote: Compressing objects: 100% (846/846), done. 2024-08-20T22:10:40.3226554Z remote: Total 1010229 (delta 1169), reused 1411 (delta 928), pack-reused 1008452 (from 1) 2024-08-20T22:11:06.0531828Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:11:06.0568270Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:06.0572720Z ##[endgroup] 2024-08-20T22:11:06.0575076Z ##[group]Determining the checkout info 2024-08-20T22:11:06.0575739Z ##[endgroup] 2024-08-20T22:11:06.0576259Z ##[group]Checking out the ref 2024-08-20T22:11:06.0577421Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:07.7781332Z ##[endgroup] 2024-08-20T22:11:07.7788562Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:07.7789938Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:07.7844176Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:07.7883402Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:07.7922706Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:07.7960145Z ##[endgroup] 2024-08-20T22:11:07.7960680Z ##[group]Fetching submodules 2024-08-20T22:11:07.7964051Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:07.8339397Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:07.8708022Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:07.8709507Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:07.8712681Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:07.8717029Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:07.8720985Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:07.8724377Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:07.8727845Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:07.8731241Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:07.8735179Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:07.8739099Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:07.8742651Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:07.8746328Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:07.8750369Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:07.8754372Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:07.8758218Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:07.8762481Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:07.8769940Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:07.8774216Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:07.8778279Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:07.8782831Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:07.8787261Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:07.8791882Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:07.8796259Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:07.8800907Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:07.8805548Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:07.8810614Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:07.8815234Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:07.8820151Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:07.8825043Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:07.8830358Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:07.8836384Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:07.8841929Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:07.8847013Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:07.8852567Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:07.8888562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:08.1869541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:08.3690624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:08.5653094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:08.8095079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:10.6762454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:21.9556835Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:22.3658780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:22.7863096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:23.3928074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:24.7016509Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:26.8207783Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:32.8497455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:34.2257262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:36.0101472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:37.3582056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:37.7896097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:38.1419314Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:39.3049866Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:39.6581385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:39.9781603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:41.5845075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:42.2788419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:42.9416065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:48.9823631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:50.8466405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:55.1127079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:55.3387194Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:12:04.3473293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:12:04.5179322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:12:04.7376763Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:12:05.6259722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:12:05.9218351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:12:06.5839127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:07.0597830Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:07.0748086Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:07.0865599Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:07.1180939Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:07.1628140Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:08.3450953Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:08.3738832Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:08.4268633Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:08.5400925Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:08.5804279Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:09.2053984Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:09.4943610Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:09.5893155Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:09.5912460Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:09.5915791Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:09.5919330Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:09.5922775Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:09.5926458Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:09.5959763Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:10.5854497Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:11.2497758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:13.3547418Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:14.4517795Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:14.7966011Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:14.9077705Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:15.4164474Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:15.4874638Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:15.5028794Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:15.6550509Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:15.7010805Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:15.7471126Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:15.7797151Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:15.8329801Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:15.8490561Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:15.8511093Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:15.8540716Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:29.1485913Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:29.1706101Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:29.2693474Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:29.2714755Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:29.2717763Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:29.2721369Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:29.2754672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:29.8229809Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:31.0449806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:32.2036399Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:32.2057050Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:32.2060380Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:32.2063939Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:32.2067687Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:32.2071459Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:32.2075433Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:32.2079297Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:32.2083233Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:32.2116426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:33.0744899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:33.4414830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:34.6525298Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:34.9610028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:35.5310340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:36.5910524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:44.1081525Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:44.5710859Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:44.5932380Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:44.6371577Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:44.6543224Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:44.6562465Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:44.6593964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:45.0101046Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:45.0331601Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:45.0810244Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:45.2031660Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:45.2231421Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:45.2687639Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:45.3356462Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:45.3813495Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:45.4107394Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:45.5355824Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:46.0475647Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:46.0511356Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:46.0514756Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:46.0549435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:46.5047059Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:47.3913092Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:47.4318950Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:47.5197374Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:47.5220649Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:47.5225324Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:47.5227330Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:47.5230623Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:47.5234532Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:47.5238553Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:47.5242197Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:47.5245784Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:47.5278001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:48.0324509Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:49.1366112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:49.4921480Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:55.5336349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:55.8022280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:55.9922627Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:56.2846051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:13:03.0549387Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:13:03.1024868Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:13:03.1215968Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:13:03.2457580Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:13:03.2622523Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:13:03.2805152Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:13:03.3002702Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:13:03.3022058Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:03.3025358Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:03.3059158Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:13:05.2259042Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:13:06.5874788Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:13:06.6400557Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:13:07.2827281Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:13:07.2970802Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:13:07.6150373Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:13:07.6176838Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:07.6180055Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:07.6211515Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:08.0869502Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:09.1998040Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:09.2811231Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:09.2927881Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:09.3079557Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:09.3514305Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:09.3860139Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:09.4349876Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:09.4679930Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:09.4701120Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:09.4704316Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:09.4707987Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:09.4711368Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:09.4742315Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:10.5763410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:10.8273664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:12.0288574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:13.0517857Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:13.0709017Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:13.1501526Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:13.1847731Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:13.1868515Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:13.1899211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:13.3714208Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:13.3760764Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:13.4126963Z Entering 'android/libs/fbjni' 2024-08-20T22:13:13.4184329Z Entering 'third_party/FP16' 2024-08-20T22:13:13.4236202Z Entering 'third_party/FXdiv' 2024-08-20T22:13:13.4287557Z Entering 'third_party/NNPACK' 2024-08-20T22:13:13.4339704Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:13.4391233Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:13.4459883Z Entering 'third_party/benchmark' 2024-08-20T22:13:13.4510956Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:13.4562694Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:13.4614690Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:13.4666575Z Entering 'third_party/cutlass' 2024-08-20T22:13:13.4725466Z Entering 'third_party/eigen' 2024-08-20T22:13:13.4779326Z Entering 'third_party/fbgemm' 2024-08-20T22:13:13.4831397Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:13.4887145Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:13.4938761Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:13.4998904Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:13.5050598Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:13.5103535Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:13.5158011Z Entering 'third_party/fmt' 2024-08-20T22:13:13.5208322Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:13.5259644Z Entering 'third_party/gloo' 2024-08-20T22:13:13.5310853Z Entering 'third_party/googletest' 2024-08-20T22:13:13.5361544Z Entering 'third_party/ideep' 2024-08-20T22:13:13.5409611Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:13.5468388Z Entering 'third_party/ittapi' 2024-08-20T22:13:13.5518278Z Entering 'third_party/kineto' 2024-08-20T22:13:13.5568037Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:13.5617349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:13.5669557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:13.5719410Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:13.5769354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:13.5818082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:13.5878216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:13.5929363Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:13.5981172Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:13.6031939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:13.6086287Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:13.6137226Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:13.6194682Z Entering 'third_party/mimalloc' 2024-08-20T22:13:13.6245473Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:13.6295977Z Entering 'third_party/nlohmann' 2024-08-20T22:13:13.6347595Z Entering 'third_party/onnx' 2024-08-20T22:13:13.6411990Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:13.6463238Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:13.6520270Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:13.6572344Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:13.6622221Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:13.6673443Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:13.6722278Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:13.6772933Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:13.6822531Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:13.6873844Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:13.6921841Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:13.6978178Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:13.7031623Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:13.7110995Z Entering 'third_party/pocketfft' 2024-08-20T22:13:13.7163902Z Entering 'third_party/protobuf' 2024-08-20T22:13:13.7218620Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:13.7268741Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:13.7326109Z Entering 'third_party/psimd' 2024-08-20T22:13:13.7379741Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:13.7431328Z Entering 'third_party/pybind11' 2024-08-20T22:13:13.7484007Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:13.7534573Z Entering 'third_party/sleef' 2024-08-20T22:13:13.7590660Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:13.7644142Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:13.7695004Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:13.7747245Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:13.7800846Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:13.7849847Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:13.7925711Z ##[endgroup] 2024-08-20T22:13:13.7926397Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:13.7928014Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:13.8296424Z Entering 'android/libs/fbjni' 2024-08-20T22:13:13.8364412Z Entering 'third_party/FP16' 2024-08-20T22:13:13.8437472Z Entering 'third_party/FXdiv' 2024-08-20T22:13:13.8509478Z Entering 'third_party/NNPACK' 2024-08-20T22:13:13.8579625Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:13.8646567Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:13.8730664Z Entering 'third_party/benchmark' 2024-08-20T22:13:13.8798672Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:13.8866910Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:13.8935738Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:13.9007342Z Entering 'third_party/cutlass' 2024-08-20T22:13:13.9082107Z Entering 'third_party/eigen' 2024-08-20T22:13:13.9151878Z Entering 'third_party/fbgemm' 2024-08-20T22:13:13.9218880Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:13.9285305Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:13.9352885Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:13.9425978Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:13.9496743Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:13.9567776Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:13.9639917Z Entering 'third_party/fmt' 2024-08-20T22:13:13.9711054Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:13.9782254Z Entering 'third_party/gloo' 2024-08-20T22:13:13.9851116Z Entering 'third_party/googletest' 2024-08-20T22:13:13.9918547Z Entering 'third_party/ideep' 2024-08-20T22:13:13.9983665Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:14.0057735Z Entering 'third_party/ittapi' 2024-08-20T22:13:14.0130065Z Entering 'third_party/kineto' 2024-08-20T22:13:14.0196559Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:14.0262990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:14.0331431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:14.0403328Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:14.0470958Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:14.0535720Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:14.0607377Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:14.0678493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:14.0746572Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:14.0819303Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:14.0893079Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:14.0961383Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:14.1030742Z Entering 'third_party/mimalloc' 2024-08-20T22:13:14.1098534Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:14.1166871Z Entering 'third_party/nlohmann' 2024-08-20T22:13:14.1240237Z Entering 'third_party/onnx' 2024-08-20T22:13:14.1320992Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:14.1389052Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:14.1461817Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:14.1530027Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:14.1596950Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:14.1663744Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:14.1730283Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:14.1798827Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:14.1865857Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:14.1933293Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:14.1997956Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:14.2066784Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:14.2143267Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:14.2233993Z Entering 'third_party/pocketfft' 2024-08-20T22:13:14.2306567Z Entering 'third_party/protobuf' 2024-08-20T22:13:14.2376879Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:14.2448851Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:14.2519477Z Entering 'third_party/psimd' 2024-08-20T22:13:14.2589964Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:14.2662185Z Entering 'third_party/pybind11' 2024-08-20T22:13:14.2731133Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:14.2799674Z Entering 'third_party/sleef' 2024-08-20T22:13:14.2869046Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:14.2937034Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:14.3004178Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:14.3073100Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:14.3139748Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:14.3205695Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:14.3300765Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:14.3668283Z Entering 'android/libs/fbjni' 2024-08-20T22:13:14.3730172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:14.3752369Z Entering 'third_party/FP16' 2024-08-20T22:13:14.3814260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:14.3837859Z Entering 'third_party/FXdiv' 2024-08-20T22:13:14.3900572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:14.3922231Z Entering 'third_party/NNPACK' 2024-08-20T22:13:14.3984046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:14.4006216Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:14.4070493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:14.4090790Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:14.4154072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:14.4190859Z Entering 'third_party/benchmark' 2024-08-20T22:13:14.4264588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:14.4276962Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:14.4341002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:14.4362938Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:14.4424510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:14.4448547Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:14.4509823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:14.4531216Z Entering 'third_party/cutlass' 2024-08-20T22:13:14.4593956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:14.4622925Z Entering 'third_party/eigen' 2024-08-20T22:13:14.4685727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:14.4709912Z Entering 'third_party/fbgemm' 2024-08-20T22:13:14.4773575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:14.4793939Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:14.4855579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:14.4876277Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:14.4943346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:14.4964863Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:14.5028736Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:14.5059160Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:14.5119734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:14.5140511Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:14.5204257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:14.5227634Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:14.5291427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:14.5315920Z Entering 'third_party/fmt' 2024-08-20T22:13:14.5378397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:14.5400272Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:14.5463179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:14.5486660Z Entering 'third_party/gloo' 2024-08-20T22:13:14.5548230Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:14.5569664Z Entering 'third_party/googletest' 2024-08-20T22:13:14.5631697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:14.5654914Z Entering 'third_party/ideep' 2024-08-20T22:13:14.5716047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:14.5735909Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:14.5797565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:14.5827476Z Entering 'third_party/ittapi' 2024-08-20T22:13:14.5890306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:14.5911629Z Entering 'third_party/kineto' 2024-08-20T22:13:14.5974514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:14.5995191Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:14.6060014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:14.6078665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:14.6141673Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:14.6163686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:14.6226934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:14.6248831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:14.6310976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:14.6332039Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:14.6395819Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:14.6413999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:14.6477419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:14.6503490Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:14.6568621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:14.6589143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:14.6652231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:14.6673243Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:14.6736385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:14.6759293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:14.6822327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:14.6847694Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:14.6909156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:14.6929977Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:14.6993503Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:14.7015281Z Entering 'third_party/mimalloc' 2024-08-20T22:13:14.7083034Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:14.7104839Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:14.7166954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:14.7188533Z Entering 'third_party/nlohmann' 2024-08-20T22:13:14.7249899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:14.7272940Z Entering 'third_party/onnx' 2024-08-20T22:13:14.7336217Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:14.7371605Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:14.7436328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:14.7456729Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:14.7523089Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:14.7549213Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:14.7612804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:14.7634903Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:14.7695952Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:14.7716183Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:14.7780290Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:14.7802525Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:14.7864395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:14.7884248Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:14.7945531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:14.7966413Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:14.8027434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:14.8049273Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:14.8110276Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:14.8129638Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:14.8191825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:14.8209570Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:14.8271708Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:14.8293441Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:14.8354914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:14.8377658Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:14.8438795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:14.8479537Z Entering 'third_party/pocketfft' 2024-08-20T22:13:14.8544117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:14.8565762Z Entering 'third_party/protobuf' 2024-08-20T22:13:14.8628900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:14.8653678Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:14.8713504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:14.8733343Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:14.8796183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:14.8821552Z Entering 'third_party/psimd' 2024-08-20T22:13:14.8884012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:14.8905751Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:14.8973010Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:14.8997287Z Entering 'third_party/pybind11' 2024-08-20T22:13:14.9059771Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:14.9082164Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:14.9144516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:14.9166484Z Entering 'third_party/sleef' 2024-08-20T22:13:14.9234063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:14.9257801Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:14.9323698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:14.9345001Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:14.9405764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:14.9426087Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:14.9486928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:14.9507802Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:14.9569651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:14.9588741Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:14.9650213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:14.9668428Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:14.9730696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:15.0657408Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:15.1026305Z Entering 'android/libs/fbjni' 2024-08-20T22:13:15.1080050Z Entering 'third_party/FP16' 2024-08-20T22:13:15.1134552Z Entering 'third_party/FXdiv' 2024-08-20T22:13:15.1185762Z Entering 'third_party/NNPACK' 2024-08-20T22:13:15.1244576Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:15.1296781Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:15.1369936Z Entering 'third_party/benchmark' 2024-08-20T22:13:15.1421198Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:15.1472886Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:15.1529596Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:15.1581971Z Entering 'third_party/cutlass' 2024-08-20T22:13:15.1642730Z Entering 'third_party/eigen' 2024-08-20T22:13:15.1695496Z Entering 'third_party/fbgemm' 2024-08-20T22:13:15.1750896Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:15.1803983Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:15.1854802Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:15.1915190Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:15.1967237Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:15.2024483Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:15.2080751Z Entering 'third_party/fmt' 2024-08-20T22:13:15.2132562Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:15.2184778Z Entering 'third_party/gloo' 2024-08-20T22:13:15.2240129Z Entering 'third_party/googletest' 2024-08-20T22:13:15.2292787Z Entering 'third_party/ideep' 2024-08-20T22:13:15.2343943Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:15.2402636Z Entering 'third_party/ittapi' 2024-08-20T22:13:15.2456917Z Entering 'third_party/kineto' 2024-08-20T22:13:15.2510233Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:15.2561968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:15.2614055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:15.2664234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:15.2714285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:15.2763064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:15.2819252Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:15.2870889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:15.2919018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:15.2970276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:15.3022862Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:15.3074071Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:15.3127488Z Entering 'third_party/mimalloc' 2024-08-20T22:13:15.3182708Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:15.3236493Z Entering 'third_party/nlohmann' 2024-08-20T22:13:15.3290797Z Entering 'third_party/onnx' 2024-08-20T22:13:15.3357392Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:15.3409025Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:15.3469738Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:15.3523740Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:15.3573629Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:15.3623990Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:15.3674696Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:15.3725242Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:15.3781669Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:15.3832650Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:15.3884762Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:15.3938429Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:15.3992059Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:15.4066865Z Entering 'third_party/pocketfft' 2024-08-20T22:13:15.4120383Z Entering 'third_party/protobuf' 2024-08-20T22:13:15.4176804Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:15.4231528Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:15.4286453Z Entering 'third_party/psimd' 2024-08-20T22:13:15.4341110Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:15.4392394Z Entering 'third_party/pybind11' 2024-08-20T22:13:15.4444746Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:15.4498546Z Entering 'third_party/sleef' 2024-08-20T22:13:15.4552990Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:15.4604465Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:15.4659949Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:15.4709851Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:15.4763316Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:15.4811753Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:15.4888538Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:15.5257092Z Entering 'android/libs/fbjni' 2024-08-20T22:13:15.5308690Z Entering 'third_party/FP16' 2024-08-20T22:13:15.5361375Z Entering 'third_party/FXdiv' 2024-08-20T22:13:15.5417872Z Entering 'third_party/NNPACK' 2024-08-20T22:13:15.5471155Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:15.5529114Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:15.5597907Z Entering 'third_party/benchmark' 2024-08-20T22:13:15.5651029Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:15.5703627Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:15.5758324Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:15.5810552Z Entering 'third_party/cutlass' 2024-08-20T22:13:15.5870607Z Entering 'third_party/eigen' 2024-08-20T22:13:15.5923225Z Entering 'third_party/fbgemm' 2024-08-20T22:13:15.5974862Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:15.6032986Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:15.6084101Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:15.6141561Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:15.6192008Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:15.6245102Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:15.6299167Z Entering 'third_party/fmt' 2024-08-20T22:13:15.6351121Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:15.6404053Z Entering 'third_party/gloo' 2024-08-20T22:13:15.6455888Z Entering 'third_party/googletest' 2024-08-20T22:13:15.6507480Z Entering 'third_party/ideep' 2024-08-20T22:13:15.6557593Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:15.6615375Z Entering 'third_party/ittapi' 2024-08-20T22:13:15.6667765Z Entering 'third_party/kineto' 2024-08-20T22:13:15.6719316Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:15.6769575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:15.6821923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:15.6872887Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:15.6922505Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:15.6974995Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:15.7032710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:15.7083122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:15.7133207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:15.7184330Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:15.7237595Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:15.7288697Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:15.7342859Z Entering 'third_party/mimalloc' 2024-08-20T22:13:15.7396696Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:15.7449522Z Entering 'third_party/nlohmann' 2024-08-20T22:13:15.7503620Z Entering 'third_party/onnx' 2024-08-20T22:13:15.7570636Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:15.7621897Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:15.7679364Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:15.7733339Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:15.7785923Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:15.7836054Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:15.7887195Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:15.7944523Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:15.7994113Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:15.8043771Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:15.8093712Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:15.8153891Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:15.8208638Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:15.8281270Z Entering 'third_party/pocketfft' 2024-08-20T22:13:15.8336566Z Entering 'third_party/protobuf' 2024-08-20T22:13:15.8393036Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:15.8444968Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:15.8501583Z Entering 'third_party/psimd' 2024-08-20T22:13:15.8556403Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:15.8610283Z Entering 'third_party/pybind11' 2024-08-20T22:13:15.8665310Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:15.8719874Z Entering 'third_party/sleef' 2024-08-20T22:13:15.8775802Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:15.8830389Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:15.8884969Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:15.8936494Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:15.8988501Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:15.9040479Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:15.9115016Z ##[endgroup] 2024-08-20T22:13:15.9166039Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:15.9202493Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:15.9405792Z Prepare all required actions 2024-08-20T22:13:15.9406241Z Getting action download info 2024-08-20T22:13:16.0737103Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:16.0737527Z env: 2024-08-20T22:13:16.0737799Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:16.0738133Z ##[endgroup] 2024-08-20T22:13:16.0808997Z ##[group]Run set -euo pipefail 2024-08-20T22:13:16.0809434Z set -euo pipefail 2024-08-20T22:13:16.0810048Z function get_ec2_metadata() { 2024-08-20T22:13:16.0810590Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:16.0811539Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:16.0812302Z  category=$1 2024-08-20T22:13:16.0812851Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:16.0813485Z  runner_name_str=i-0035567a13af53524 2024-08-20T22:13:16.0813976Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:16.0814666Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:16.0815305Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:16.0816171Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:16.0816785Z  else 2024-08-20T22:13:16.0817454Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:16.0818038Z  fi 2024-08-20T22:13:16.0818324Z } 2024-08-20T22:13:16.0818685Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:16.0819283Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:16.0819943Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:16.0820526Z echo "system info $(uname -a)" 2024-08-20T22:13:16.0830810Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:16.0831296Z env: 2024-08-20T22:13:16.0831571Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:16.0831912Z ##[endgroup] 2024-08-20T22:13:16.0936781Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:16.0992862Z instance-id: i-0035567a13af53524 2024-08-20T22:13:16.1063994Z instance-type: g5.4xlarge 2024-08-20T22:13:16.1078083Z system info Linux ip-10-0-45-195.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:16.1103215Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:16.1104124Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:16.1113346Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:16.1114072Z env: 2024-08-20T22:13:16.1114350Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:16.1114695Z ##[endgroup] 2024-08-20T22:13:16.1198183Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:16.1198792Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:16.1199326Z  echo "Docker daemon is running..."; 2024-08-20T22:13:16.1199776Z else 2024-08-20T22:13:16.1200262Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:16.1200842Z fi 2024-08-20T22:13:16.1209543Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:16.1210038Z env: 2024-08-20T22:13:16.1210440Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:16.1210777Z ##[endgroup] 2024-08-20T22:13:16.1293649Z Docker daemon is running... 2024-08-20T22:13:16.1345247Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:16.1345815Z with: 2024-08-20T22:13:16.1346096Z shell: bash 2024-08-20T22:13:16.1346380Z timeout_minutes: 5 2024-08-20T22:13:16.1346698Z max_attempts: 3 2024-08-20T22:13:16.1347010Z retry_wait_seconds: 30 2024-08-20T22:13:16.1350316Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:16.1353328Z polling_interval_seconds: 1 2024-08-20T22:13:16.1353698Z warning_on_retry: true 2024-08-20T22:13:16.1354037Z continue_on_error: false 2024-08-20T22:13:16.1354369Z env: 2024-08-20T22:13:16.1354638Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:16.1354984Z AWS_RETRY_MODE: standard 2024-08-20T22:13:16.1355324Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:16.1355658Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:16.1356028Z ##[endgroup] 2024-08-20T22:13:17.3118198Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:17.3119356Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:17.3120494Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:17.3121200Z 2024-08-20T22:13:17.3121353Z Login Succeeded 2024-08-20T22:13:18.1973853Z Command completed after 1 attempt(s). 2024-08-20T22:13:18.2040036Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:18.2040755Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:18.2041398Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:18.2050443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:18.2050935Z env: 2024-08-20T22:13:18.2051215Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:18.2051563Z ##[endgroup] 2024-08-20T22:13:18.2148069Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:18.2148840Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:18.2149430Z # shellcheck disable=SC2046 2024-08-20T22:13:18.2149883Z docker stop $(docker ps -q) || true 2024-08-20T22:13:18.2150353Z # Prune all of the docker images 2024-08-20T22:13:18.2150799Z docker system prune -af 2024-08-20T22:13:18.2159627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:18.2160119Z env: 2024-08-20T22:13:18.2160386Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:18.2160728Z ##[endgroup] 2024-08-20T22:13:18.2475166Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:18.2475988Z See 'docker stop --help'. 2024-08-20T22:13:18.2478471Z 2024-08-20T22:13:18.2478881Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:18.2479339Z 2024-08-20T22:13:18.2479501Z Stop one or more running containers 2024-08-20T22:13:18.2656737Z Total reclaimed space: 0B 2024-08-20T22:13:18.2701277Z ##[group]Run set +e 2024-08-20T22:13:18.2701632Z set +e 2024-08-20T22:13:18.2701931Z set -x 2024-08-20T22:13:18.2702225Z  2024-08-20T22:13:18.2702536Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:18.2703321Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:18.2704399Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:18.2705146Z # one is returned at random 2024-08-20T22:13:18.2705685Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:18.2706189Z  2024-08-20T22:13:18.2706514Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:18.2707133Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:18.2708346Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:18.2708952Z  2024-08-20T22:13:18.2709284Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:18.2709845Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:18.2710361Z  exit 1 2024-08-20T22:13:18.2710674Z  fi 2024-08-20T22:13:18.2710962Z fi 2024-08-20T22:13:18.2711239Z  2024-08-20T22:13:18.2711596Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:18.2712113Z  # Clean up any old records first 2024-08-20T22:13:18.2712625Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:18.2713080Z fi 2024-08-20T22:13:18.2713351Z  2024-08-20T22:13:18.2713784Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:18.2714338Z cat /etc/hosts 2024-08-20T22:13:18.2723329Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:18.2723821Z env: 2024-08-20T22:13:18.2724102Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:18.2724454Z ##[endgroup] 2024-08-20T22:13:18.2752611Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:18.2758888Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:18.2759352Z ++ tail -n1 2024-08-20T22:13:18.2924058Z + RESOLVED_IP=18.160.10.76 2024-08-20T22:13:18.2924543Z + '[' -z 18.160.10.76 ']' 2024-08-20T22:13:18.2924996Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:18.2938557Z 18.165.98.47 download.pytorch.org 2024-08-20T22:13:18.2940666Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T22:13:18.4237076Z + echo '18.160.10.76 download.pytorch.org' 2024-08-20T22:13:18.4237612Z + sudo tee -a /etc/hosts 2024-08-20T22:13:18.4561341Z 18.160.10.76 download.pytorch.org 2024-08-20T22:13:18.4583555Z + cat /etc/hosts 2024-08-20T22:13:18.4593683Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:18.4606525Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:18.4607084Z 18.160.10.76 download.pytorch.org 2024-08-20T22:13:18.4763662Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:18.4764270Z with: 2024-08-20T22:13:18.4765376Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4766605Z docker-build-dir: .ci/docker 2024-08-20T22:13:18.4766984Z working-directory: . 2024-08-20T22:13:18.4767447Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:18.4767971Z force-push: false 2024-08-20T22:13:18.4768270Z env: 2024-08-20T22:13:18.4768545Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:18.4768882Z ##[endgroup] 2024-08-20T22:13:18.4787925Z ##[group]Run set -ex 2024-08-20T22:13:18.4788286Z set -ex 2024-08-20T22:13:18.4788577Z  2024-08-20T22:13:18.4789142Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:18.4790129Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:18.4790927Z # job could then download the pre-built image as usual 2024-08-20T22:13:18.4791676Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:18.4792342Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4792974Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4793533Z  2024-08-20T22:13:18.4794040Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:18.4794659Z  exit 0 2024-08-20T22:13:18.4794968Z else 2024-08-20T22:13:18.4795328Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4795779Z fi 2024-08-20T22:13:18.4796054Z  2024-08-20T22:13:18.4796523Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:18.4797369Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:18.4798152Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:18.4798862Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:18.4799586Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4800283Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4800838Z else 2024-08-20T22:13:18.4801270Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:18.4801938Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4802847Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4803596Z fi 2024-08-20T22:13:18.4813076Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:18.4813576Z env: 2024-08-20T22:13:18.4813855Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:18.4814198Z REPO_NAME: pytorch 2024-08-20T22:13:18.4815328Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4816544Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:18.4817038Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:18.4817546Z ##[endgroup] 2024-08-20T22:13:18.4848114Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:18.4848673Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:18.4849160Z + echo skip=false 2024-08-20T22:13:18.4851363Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:18.4856021Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4857866Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:18.4883516Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4884695Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4886610Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4921641Z ##[group]Run set +e 2024-08-20T22:13:18.4922309Z set +e 2024-08-20T22:13:18.4922765Z set -x 2024-08-20T22:13:18.4923120Z  2024-08-20T22:13:18.4923524Z login() { 2024-08-20T22:13:18.4924341Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:18.4925157Z } 2024-08-20T22:13:18.4925607Z  2024-08-20T22:13:18.4926043Z retry () { 2024-08-20T22:13:18.4926564Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:18.4927072Z } 2024-08-20T22:13:18.4927492Z  2024-08-20T22:13:18.4927931Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:18.4928449Z  2024-08-20T22:13:18.4929030Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:18.4929869Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:18.4930478Z  exit 0 2024-08-20T22:13:18.4930929Z fi 2024-08-20T22:13:18.4931372Z  2024-08-20T22:13:18.4931973Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:18.4932979Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:18.4934071Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:18.4934805Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:18.4935601Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:18.4936150Z else 2024-08-20T22:13:18.4936726Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:18.4937597Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:18.4938210Z fi 2024-08-20T22:13:18.4938559Z  2024-08-20T22:13:18.4939055Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:18.4939742Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4940387Z  2024-08-20T22:13:18.4941110Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:18.4941971Z  exit 0 2024-08-20T22:13:18.4942451Z fi 2024-08-20T22:13:18.4942839Z  2024-08-20T22:13:18.4943348Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:18.4944518Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:18.4945439Z  exit 1 2024-08-20T22:13:18.4945863Z fi 2024-08-20T22:13:18.4946260Z  2024-08-20T22:13:18.4946928Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:18.4948121Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:18.4949192Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:18.4950301Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:18.4951491Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:18.4952467Z fi 2024-08-20T22:13:18.4952875Z  2024-08-20T22:13:18.4953329Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:18.4961798Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:18.4962391Z env: 2024-08-20T22:13:18.4962918Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:18.4963380Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:18.4963904Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:18.4965365Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4966716Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:18.4967624Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:18.4968201Z ##[endgroup] 2024-08-20T22:13:18.4999831Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:18.5000825Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:18.5002330Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:18.5004819Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:19.0375889Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:19.0376849Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:19.0378146Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:19.0378714Z 2024-08-20T22:13:19.0379002Z Login Succeeded 2024-08-20T22:13:19.0401498Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:19.2583790Z { 2024-08-20T22:13:19.2584661Z "schemaVersion": 2, 2024-08-20T22:13:19.2585711Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:19.2587102Z "config": { 2024-08-20T22:13:19.2587841Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:19.2589331Z "size": 48711, 2024-08-20T22:13:19.2590345Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:19.2591281Z }, 2024-08-20T22:13:19.2591790Z "layers": [ 2024-08-20T22:13:19.2592184Z { 2024-08-20T22:13:19.2593086Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2594181Z "size": 28580681, 2024-08-20T22:13:19.2594807Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:19.2596258Z }, 2024-08-20T22:13:19.2596694Z { 2024-08-20T22:13:19.2597404Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2598073Z "size": 7943451, 2024-08-20T22:13:19.2598944Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:19.2600092Z }, 2024-08-20T22:13:19.2600604Z { 2024-08-20T22:13:19.2601408Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2602328Z "size": 55728572, 2024-08-20T22:13:19.2603788Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:19.2605050Z }, 2024-08-20T22:13:19.2605497Z { 2024-08-20T22:13:19.2606210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2607508Z "size": 186, 2024-08-20T22:13:19.2608147Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:19.2608884Z }, 2024-08-20T22:13:19.2609333Z { 2024-08-20T22:13:19.2609902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2610526Z "size": 6886, 2024-08-20T22:13:19.2611237Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:19.2612194Z }, 2024-08-20T22:13:19.2612881Z { 2024-08-20T22:13:19.2613441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2614150Z "size": 1329001756, 2024-08-20T22:13:19.2614966Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:19.2615774Z }, 2024-08-20T22:13:19.2616100Z { 2024-08-20T22:13:19.2616772Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2617474Z "size": 62501, 2024-08-20T22:13:19.2618129Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:19.2619244Z }, 2024-08-20T22:13:19.2619714Z { 2024-08-20T22:13:19.2620284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2621121Z "size": 1684, 2024-08-20T22:13:19.2621822Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:19.2622566Z }, 2024-08-20T22:13:19.2622925Z { 2024-08-20T22:13:19.2623489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2624197Z "size": 1523, 2024-08-20T22:13:19.2624851Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:19.2625677Z }, 2024-08-20T22:13:19.2625997Z { 2024-08-20T22:13:19.2626545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2627411Z "size": 2528295403, 2024-08-20T22:13:19.2628077Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:19.2628794Z }, 2024-08-20T22:13:19.2629244Z { 2024-08-20T22:13:19.2629780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2630404Z "size": 86016, 2024-08-20T22:13:19.2631173Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:19.2631906Z }, 2024-08-20T22:13:19.2632291Z { 2024-08-20T22:13:19.2632852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2633909Z "size": 1822, 2024-08-20T22:13:19.2634599Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:19.2635404Z }, 2024-08-20T22:13:19.2635728Z { 2024-08-20T22:13:19.2636284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2637005Z "size": 246761563, 2024-08-20T22:13:19.2637647Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:19.2638407Z }, 2024-08-20T22:13:19.2638838Z { 2024-08-20T22:13:19.2639394Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2640013Z "size": 544, 2024-08-20T22:13:19.2640705Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:19.2641436Z }, 2024-08-20T22:13:19.2641838Z { 2024-08-20T22:13:19.2642332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2643017Z "size": 1258, 2024-08-20T22:13:19.2643707Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:19.2644429Z }, 2024-08-20T22:13:19.2644757Z { 2024-08-20T22:13:19.2645368Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2646047Z "size": 484, 2024-08-20T22:13:19.2646635Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:19.2647384Z }, 2024-08-20T22:13:19.2647756Z { 2024-08-20T22:13:19.2648290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2648958Z "size": 91711766, 2024-08-20T22:13:19.2649638Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:19.2650369Z }, 2024-08-20T22:13:19.2650774Z { 2024-08-20T22:13:19.2651291Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2651954Z "size": 3390, 2024-08-20T22:13:19.2652858Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:19.2653554Z }, 2024-08-20T22:13:19.2653906Z { 2024-08-20T22:13:19.2654524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2655182Z "size": 1909, 2024-08-20T22:13:19.2655777Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:19.2656573Z }, 2024-08-20T22:13:19.2656928Z { 2024-08-20T22:13:19.2657484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2658158Z "size": 701, 2024-08-20T22:13:19.2658807Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:19.2659721Z }, 2024-08-20T22:13:19.2660156Z { 2024-08-20T22:13:19.2660695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2661374Z "size": 2864254879, 2024-08-20T22:13:19.2662111Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:19.2662827Z }, 2024-08-20T22:13:19.2663176Z { 2024-08-20T22:13:19.2663764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2664437Z "size": 380, 2024-08-20T22:13:19.2665036Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:19.2665804Z }, 2024-08-20T22:13:19.2666195Z { 2024-08-20T22:13:19.2666786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2667500Z "size": 12900, 2024-08-20T22:13:19.2668181Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:19.2668971Z }, 2024-08-20T22:13:19.2669303Z { 2024-08-20T22:13:19.2669843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2670570Z "size": 803, 2024-08-20T22:13:19.2671260Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:19.2671928Z }, 2024-08-20T22:13:19.2672324Z { 2024-08-20T22:13:19.2672871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2673550Z "size": 106, 2024-08-20T22:13:19.2674195Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:19.2674936Z }, 2024-08-20T22:13:19.2675288Z { 2024-08-20T22:13:19.2675862Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2676526Z "size": 503, 2024-08-20T22:13:19.2677171Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:19.2677973Z }, 2024-08-20T22:13:19.2678305Z { 2024-08-20T22:13:19.2678850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2679594Z "size": 121477267, 2024-08-20T22:13:19.2680289Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:19.2680969Z }, 2024-08-20T22:13:19.2682823Z { 2024-08-20T22:13:19.2683364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2684047Z "size": 109, 2024-08-20T22:13:19.2684730Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:19.2685454Z }, 2024-08-20T22:13:19.2685831Z { 2024-08-20T22:13:19.2686377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2687040Z "size": 490, 2024-08-20T22:13:19.2687695Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:19.2688456Z }, 2024-08-20T22:13:19.2688795Z { 2024-08-20T22:13:19.2689359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2690064Z "size": 297, 2024-08-20T22:13:19.2690726Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:19.2691403Z }, 2024-08-20T22:13:19.2691911Z { 2024-08-20T22:13:19.2692469Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2693180Z "size": 103, 2024-08-20T22:13:19.2693815Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:19.2694545Z }, 2024-08-20T22:13:19.2694950Z { 2024-08-20T22:13:19.2695489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2696144Z "size": 1473, 2024-08-20T22:13:19.2696845Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:19.2697596Z }, 2024-08-20T22:13:19.2697914Z { 2024-08-20T22:13:19.2698494Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2699283Z "size": 594729516, 2024-08-20T22:13:19.2699942Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:19.2700691Z }, 2024-08-20T22:13:19.2701053Z { 2024-08-20T22:13:19.2701595Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2702316Z "size": 164, 2024-08-20T22:13:19.2702964Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:19.2703709Z }, 2024-08-20T22:13:19.2704117Z { 2024-08-20T22:13:19.2704613Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2705288Z "size": 564, 2024-08-20T22:13:19.2705980Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:19.2706705Z }, 2024-08-20T22:13:19.2707036Z { 2024-08-20T22:13:19.2707684Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2708364Z "size": 35874203, 2024-08-20T22:13:19.2709095Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:19.2709863Z }, 2024-08-20T22:13:19.2710242Z { 2024-08-20T22:13:19.2710780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2711455Z "size": 104, 2024-08-20T22:13:19.2712112Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:19.2712821Z }, 2024-08-20T22:13:19.2713232Z { 2024-08-20T22:13:19.2713749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2714416Z "size": 425, 2024-08-20T22:13:19.2715124Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:19.2715884Z }, 2024-08-20T22:13:19.2716221Z { 2024-08-20T22:13:19.2716827Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2717493Z "size": 20262183, 2024-08-20T22:13:19.2718125Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:19.2718922Z }, 2024-08-20T22:13:19.2719275Z { 2024-08-20T22:13:19.2719854Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2720497Z "size": 438, 2024-08-20T22:13:19.2721136Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:19.2722010Z }, 2024-08-20T22:13:19.2722375Z { 2024-08-20T22:13:19.2722900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2723638Z "size": 701, 2024-08-20T22:13:19.2724291Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:19.2725022Z }, 2024-08-20T22:13:19.2725422Z { 2024-08-20T22:13:19.2725952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2726627Z "size": 141, 2024-08-20T22:13:19.2727292Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:19.2728022Z }, 2024-08-20T22:13:19.2728388Z { 2024-08-20T22:13:19.2728981Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2729593Z "size": 135, 2024-08-20T22:13:19.2730371Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:19.2731140Z }, 2024-08-20T22:13:19.2731505Z { 2024-08-20T22:13:19.2732000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2732714Z "size": 4838068375, 2024-08-20T22:13:19.2733693Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:19.2734425Z }, 2024-08-20T22:13:19.2734853Z { 2024-08-20T22:13:19.2735418Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2736071Z "size": 194, 2024-08-20T22:13:19.2736722Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:19.2737637Z }, 2024-08-20T22:13:19.2737988Z { 2024-08-20T22:13:19.2738592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2739215Z "size": 564, 2024-08-20T22:13:19.2740052Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:19.2740856Z }, 2024-08-20T22:13:19.2741211Z { 2024-08-20T22:13:19.2741712Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2742444Z "size": 43162921, 2024-08-20T22:13:19.2743106Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:19.2743804Z }, 2024-08-20T22:13:19.2744227Z { 2024-08-20T22:13:19.2744763Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2745487Z "size": 106, 2024-08-20T22:13:19.2746084Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:19.2746810Z }, 2024-08-20T22:13:19.2747300Z { 2024-08-20T22:13:19.2747846Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2748459Z "size": 1380, 2024-08-20T22:13:19.2749181Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:19.2749906Z }, 2024-08-20T22:13:19.2750244Z { 2024-08-20T22:13:19.2750845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2751544Z "size": 701, 2024-08-20T22:13:19.2752267Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:19.2753006Z }, 2024-08-20T22:13:19.2753384Z { 2024-08-20T22:13:19.2753916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2754619Z "size": 139, 2024-08-20T22:13:19.2755237Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:19.2755954Z }, 2024-08-20T22:13:19.2756365Z { 2024-08-20T22:13:19.2756921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2757535Z "size": 118, 2024-08-20T22:13:19.2758241Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:19.2758972Z }, 2024-08-20T22:13:19.2759282Z { 2024-08-20T22:13:19.2759886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2760555Z "size": 4324672728, 2024-08-20T22:13:19.2761282Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:19.2761983Z }, 2024-08-20T22:13:19.2762326Z { 2024-08-20T22:13:19.2762902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2763575Z "size": 174, 2024-08-20T22:13:19.2764163Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:19.2764938Z }, 2024-08-20T22:13:19.2765300Z { 2024-08-20T22:13:19.2765804Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2766528Z "size": 907, 2024-08-20T22:13:19.2767183Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:19.2767927Z }, 2024-08-20T22:13:19.2768440Z { 2024-08-20T22:13:19.2768967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2769636Z "size": 701, 2024-08-20T22:13:19.2770339Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:19.2771031Z }, 2024-08-20T22:13:19.2771402Z { 2024-08-20T22:13:19.2772024Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2772677Z "size": 134, 2024-08-20T22:13:19.2773303Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:19.2774079Z }, 2024-08-20T22:13:19.2774428Z { 2024-08-20T22:13:19.2774944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2775748Z "size": 32, 2024-08-20T22:13:19.2776418Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:19.2777138Z }, 2024-08-20T22:13:19.2777496Z { 2024-08-20T22:13:19.2778063Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2778750Z "size": 158, 2024-08-20T22:13:19.2779462Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:19.2780133Z }, 2024-08-20T22:13:19.2780475Z { 2024-08-20T22:13:19.2781073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2781725Z "size": 1841, 2024-08-20T22:13:19.2782327Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:19.2783131Z }, 2024-08-20T22:13:19.2783475Z { 2024-08-20T22:13:19.2783968Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2784704Z "size": 7529776, 2024-08-20T22:13:19.2785386Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:19.2786177Z }, 2024-08-20T22:13:19.2786485Z { 2024-08-20T22:13:19.2787018Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2787800Z "size": 163, 2024-08-20T22:13:19.2788434Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:19.2789101Z }, 2024-08-20T22:13:19.2789531Z { 2024-08-20T22:13:19.2790070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2790693Z "size": 7943, 2024-08-20T22:13:19.2791417Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:19.2792147Z }, 2024-08-20T22:13:19.2792547Z { 2024-08-20T22:13:19.2793088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2793751Z "size": 8068, 2024-08-20T22:13:19.2794423Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:19.2795206Z }, 2024-08-20T22:13:19.2795522Z { 2024-08-20T22:13:19.2796080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2796807Z "size": 302, 2024-08-20T22:13:19.2797446Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:19.2798144Z }, 2024-08-20T22:13:19.2798555Z { 2024-08-20T22:13:19.2799150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2799770Z "size": 7629638, 2024-08-20T22:13:19.2800470Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:19.2801208Z }, 2024-08-20T22:13:19.2801554Z { 2024-08-20T22:13:19.2802101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2802770Z "size": 108, 2024-08-20T22:13:19.2803415Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:19.2804204Z }, 2024-08-20T22:13:19.2804538Z { 2024-08-20T22:13:19.2805076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2805806Z "size": 54145661, 2024-08-20T22:13:19.2806576Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:19.2807311Z }, 2024-08-20T22:13:19.2807728Z { 2024-08-20T22:13:19.2808260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2808877Z "size": 473, 2024-08-20T22:13:19.2809597Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:19.2810329Z }, 2024-08-20T22:13:19.2810798Z { 2024-08-20T22:13:19.2811302Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2811949Z "size": 1374858766, 2024-08-20T22:13:19.2812686Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:19.2813538Z }, 2024-08-20T22:13:19.2813848Z { 2024-08-20T22:13:19.2814453Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2815096Z "size": 106, 2024-08-20T22:13:19.2815732Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:19.2816526Z }, 2024-08-20T22:13:19.2816880Z { 2024-08-20T22:13:19.2817442Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2818129Z "size": 559, 2024-08-20T22:13:19.2818766Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:19.2819511Z }, 2024-08-20T22:13:19.2819951Z { 2024-08-20T22:13:19.2820498Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2821182Z "size": 46248488, 2024-08-20T22:13:19.2821896Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:19.2822617Z }, 2024-08-20T22:13:19.2822964Z { 2024-08-20T22:13:19.2823549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2824224Z "size": 111, 2024-08-20T22:13:19.2824839Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:19.2825600Z }, 2024-08-20T22:13:19.2825974Z { 2024-08-20T22:13:19.2826518Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2827216Z "size": 32, 2024-08-20T22:13:19.2827955Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:19.2828685Z }, 2024-08-20T22:13:19.2829113Z { 2024-08-20T22:13:19.2829610Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2830274Z "size": 32, 2024-08-20T22:13:19.2830986Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:19.2831673Z }, 2024-08-20T22:13:19.2832027Z { 2024-08-20T22:13:19.2832637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2833297Z "size": 32, 2024-08-20T22:13:19.2834275Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:19.2835110Z }, 2024-08-20T22:13:19.2835468Z { 2024-08-20T22:13:19.2836076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:19.2836703Z "size": 32, 2024-08-20T22:13:19.2837342Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:19.2838150Z } 2024-08-20T22:13:19.2838464Z ] 2024-08-20T22:13:19.2838810Z } 2024-08-20T22:13:19.2839266Z + exit 0 2024-08-20T22:13:19.3158048Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:19.3158800Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:19.3159571Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:19.3169204Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:19.3169715Z env: 2024-08-20T22:13:19.3169984Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.3171147Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:19.3172331Z ##[endgroup] 2024-08-20T22:13:19.3204241Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:19.3259628Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:19.3260225Z with: 2024-08-20T22:13:19.3261309Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:19.3262624Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:19.3263144Z env: 2024-08-20T22:13:19.3263596Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.3263949Z ##[endgroup] 2024-08-20T22:13:19.3281601Z ##[group]Run set -x 2024-08-20T22:13:19.3281963Z set -x 2024-08-20T22:13:19.3282280Z set +e 2024-08-20T22:13:19.3282582Z  2024-08-20T22:13:19.3282869Z login() { 2024-08-20T22:13:19.3283544Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:19.3284308Z } 2024-08-20T22:13:19.3284672Z  2024-08-20T22:13:19.3284984Z retry () { 2024-08-20T22:13:19.3285375Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:19.3285829Z } 2024-08-20T22:13:19.3286107Z  2024-08-20T22:13:19.3286418Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:19.3286827Z  2024-08-20T22:13:19.3287098Z set -e 2024-08-20T22:13:19.3287572Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:19.3288292Z # only pull docker image if it's not available locally 2024-08-20T22:13:19.3289058Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:19.3289758Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:19.3290199Z fi 2024-08-20T22:13:19.3298931Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:19.3299424Z env: 2024-08-20T22:13:19.3299698Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.3300862Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:19.3302181Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:19.3302702Z ##[endgroup] 2024-08-20T22:13:19.3331891Z + set +e 2024-08-20T22:13:19.3332429Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:19.3333073Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:19.3336370Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:19.3338052Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:19.8738724Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:19.8740022Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:19.8740917Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:19.8741549Z 2024-08-20T22:13:19.8741691Z Login Succeeded 2024-08-20T22:13:19.8762854Z + set -e 2024-08-20T22:13:19.8764225Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:19.8921860Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:19.8924045Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:20.1286094Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:20.1287412Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:20.1287898Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:20.1288725Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:20.1289279Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:20.1289813Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:20.1290349Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:20.1290886Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:20.1291324Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:20.1291749Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:20.1292257Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:20.1292751Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:20.1293507Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:20.1294020Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:20.1294533Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:20.1295039Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:20.1295562Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:20.1296034Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:20.1296494Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:20.1296877Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:20.1297268Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:20.1297762Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:20.1298181Z d527cbbb87e3: Waiting 2024-08-20T22:13:20.1298602Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:20.1299069Z b57676e46aee: Waiting 2024-08-20T22:13:20.1299515Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:20.1299911Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:20.1300331Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:20.1300780Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:20.1301168Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:20.1301547Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:20.1301921Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:20.1302293Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:20.1302656Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:20.1303023Z a8c1e85b5e14: Waiting 2024-08-20T22:13:20.1303408Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:20.1303780Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:20.1304131Z a41a8d1c11c8: Waiting 2024-08-20T22:13:20.1304471Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:20.1304837Z 75b2abb16c68: Waiting 2024-08-20T22:13:20.1305176Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:20.1305531Z ca426ffc9d2b: Waiting 2024-08-20T22:13:20.1305870Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:20.1306230Z 0bf61114d37b: Waiting 2024-08-20T22:13:20.1306555Z 019620471c25: Pulling fs layer 2024-08-20T22:13:20.1306911Z 4373aa743caf: Waiting 2024-08-20T22:13:20.1307246Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:20.1307781Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:20.1308118Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:20.1308480Z 2df849f72ba4: Waiting 2024-08-20T22:13:20.1308817Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:20.1309177Z 0c1227890755: Waiting 2024-08-20T22:13:20.1309510Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:20.1309890Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:20.1310287Z 34c47d63c5de: Waiting 2024-08-20T22:13:20.1310622Z d8d1234baab3: Waiting 2024-08-20T22:13:20.1310964Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:20.1311346Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:20.1311725Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:20.1312081Z d3547bd43146: Waiting 2024-08-20T22:13:20.1312414Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:20.1312787Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:20.1313146Z 5645520256fa: Waiting 2024-08-20T22:13:20.1313458Z 7ed32bc8e469: Waiting 2024-08-20T22:13:20.1313793Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:20.1314163Z 1fef4bd19792: Waiting 2024-08-20T22:13:20.1314490Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:20.1314853Z 6efb96fc87ea: Waiting 2024-08-20T22:13:20.1315190Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:20.1315571Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:20.1315942Z f957c1624481: Pulling fs layer 2024-08-20T22:13:20.1316319Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:20.1316682Z 7cbfce999d76: Waiting 2024-08-20T22:13:20.1317002Z 829ed7d91e38: Waiting 2024-08-20T22:13:20.1317474Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:20.1317805Z 6658e558ebe4: Waiting 2024-08-20T22:13:20.1318123Z 33444a5a7e01: Waiting 2024-08-20T22:13:20.1318435Z cbb79a705353: Waiting 2024-08-20T22:13:20.1318752Z 7c47122f4e09: Waiting 2024-08-20T22:13:20.1319069Z 1811ff4d3556: Waiting 2024-08-20T22:13:20.1319403Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:20.1319780Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:20.1320164Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:20.1320545Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:20.1320990Z 0f085d4fa279: Waiting 2024-08-20T22:13:20.1321313Z c42b3602a1fb: Waiting 2024-08-20T22:13:20.1321633Z 916c5e858016: Waiting 2024-08-20T22:13:20.1321947Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:20.1322275Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:20.1322656Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:20.1323038Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:20.1323404Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:20.1323719Z 5a4eb19d7219: Waiting 2024-08-20T22:13:20.1324050Z 11fad972ef6c: Waiting 2024-08-20T22:13:20.1324367Z 7754ee068b33: Waiting 2024-08-20T22:13:20.1324676Z 4a0412373305: Waiting 2024-08-20T22:13:20.1324992Z afc429e87b59: Waiting 2024-08-20T22:13:20.1325309Z 302c8c177944: Waiting 2024-08-20T22:13:20.1325628Z da6a838a66c7: Waiting 2024-08-20T22:13:20.1325937Z 02970347dff6: Waiting 2024-08-20T22:13:20.1326268Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:20.1326636Z 560cd6122a38: Waiting 2024-08-20T22:13:20.1326945Z 0479fbac3361: Waiting 2024-08-20T22:13:20.1327290Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:20.1327672Z ee1273153675: Pulling fs layer 2024-08-20T22:13:20.1328032Z f957c1624481: Waiting 2024-08-20T22:13:20.1328357Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:20.1328739Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:20.1329107Z 78ee77eddef2: Waiting 2024-08-20T22:13:20.1329426Z 33b639226eee: Waiting 2024-08-20T22:13:20.1329759Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:20.1330123Z a7a5392e2bf6: Waiting 2024-08-20T22:13:20.1330464Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:20.1330838Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:20.1331202Z b10de02255c0: Waiting 2024-08-20T22:13:20.1331538Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:20.1331917Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:20.1332277Z 768c31dca016: Waiting 2024-08-20T22:13:20.1332611Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:20.1332972Z 829c0e19f4b7: Waiting 2024-08-20T22:13:20.1333293Z b119c4fc6c58: Waiting 2024-08-20T22:13:20.1333816Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:20.1334138Z 83808b2eff4d: Waiting 2024-08-20T22:13:20.1334460Z 4f4fb700ef54: Waiting 2024-08-20T22:13:20.1334777Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:20.1335093Z 5d115b29b5d5: Waiting 2024-08-20T22:13:20.1335414Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:20.1335732Z e22979f9e0ee: Waiting 2024-08-20T22:13:20.1336045Z eb09cbf3e416: Waiting 2024-08-20T22:13:20.1336363Z 9752d6fbbcab: Waiting 2024-08-20T22:13:20.1336685Z d79fd7d52698: Waiting 2024-08-20T22:13:20.1336998Z 385d0421d67c: Waiting 2024-08-20T22:13:20.1337320Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:20.1337642Z f1d6874dee9c: Waiting 2024-08-20T22:13:20.1337960Z ee1273153675: Waiting 2024-08-20T22:13:20.1338361Z 019620471c25: Waiting 2024-08-20T22:13:20.1338787Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:20.1339209Z 37513e9b43d7: Waiting 2024-08-20T22:13:20.2690129Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:20.2690688Z 224fe954d725: Download complete 2024-08-20T22:13:20.3399322Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:20.3399984Z d527cbbb87e3: Download complete 2024-08-20T22:13:20.4129495Z b57676e46aee: Download complete 2024-08-20T22:13:20.4605380Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:20.4605864Z 7a2c55901189: Download complete 2024-08-20T22:13:20.5349111Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:13:20.5349694Z a41a8d1c11c8: Download complete 2024-08-20T22:13:20.6497450Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:20.6497966Z 0c1227890755: Download complete 2024-08-20T22:13:20.7294124Z d8d1234baab3: Verifying Checksum 2024-08-20T22:13:20.7294606Z d8d1234baab3: Download complete 2024-08-20T22:13:20.7702777Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:20.7703352Z 75722010b82e: Download complete 2024-08-20T22:13:20.8493557Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:20.8494188Z ec1e7978c1fe: Download complete 2024-08-20T22:13:20.9221753Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:13:20.9222272Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:21.6250633Z 7a2c55901189: Pull complete 2024-08-20T22:13:21.9512522Z 224fe954d725: Pull complete 2024-08-20T22:13:22.8730207Z 75722010b82e: Pull complete 2024-08-20T22:13:22.8921828Z d527cbbb87e3: Pull complete 2024-08-20T22:13:22.9095659Z b57676e46aee: Pull complete 2024-08-20T22:13:23.4428610Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:23.4429098Z ca426ffc9d2b: Download complete 2024-08-20T22:13:23.5249543Z 4373aa743caf: Download complete 2024-08-20T22:13:23.5999074Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:23.5999681Z 75b2abb16c68: Download complete 2024-08-20T22:13:23.6745397Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:24.6624939Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:24.6625639Z 0bf61114d37b: Download complete 2024-08-20T22:13:24.7325492Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:24.7325949Z 2df849f72ba4: Download complete 2024-08-20T22:13:24.8000595Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:24.8001020Z 34c47d63c5de: Download complete 2024-08-20T22:13:24.8779000Z 1fef4bd19792: Download complete 2024-08-20T22:13:33.7942383Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:33.7942875Z a8c1e85b5e14: Download complete 2024-08-20T22:13:33.8606964Z b119c4fc6c58: Download complete 2024-08-20T22:13:33.9381154Z 33b639226eee: Download complete 2024-08-20T22:13:34.0201474Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:34.0201945Z d3547bd43146: Download complete 2024-08-20T22:13:34.0966466Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:34.0967017Z 5645520256fa: Download complete 2024-08-20T22:13:34.2145321Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:13:34.2145773Z 829ed7d91e38: Download complete 2024-08-20T22:13:35.4747225Z 385d0421d67c: Download complete 2024-08-20T22:13:35.5521366Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:35.5521906Z 7cbfce999d76: Download complete 2024-08-20T22:13:35.6166746Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:35.6167350Z 6658e558ebe4: Download complete 2024-08-20T22:13:35.6866312Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:13:35.6866979Z 33444a5a7e01: Download complete 2024-08-20T22:13:35.7578508Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:35.7578976Z 7c47122f4e09: Download complete 2024-08-20T22:13:35.8454122Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:35.8454608Z 1811ff4d3556: Download complete 2024-08-20T22:13:41.8473804Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:41.8474317Z 7754ee068b33: Download complete 2024-08-20T22:13:41.9221957Z 0479fbac3361: Download complete 2024-08-20T22:13:41.9991536Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:13:41.9992158Z f1d6874dee9c: Download complete 2024-08-20T22:13:42.4086594Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:42.4087172Z afc429e87b59: Download complete 2024-08-20T22:13:42.4952551Z 019620471c25: Verifying Checksum 2024-08-20T22:13:42.4953122Z 019620471c25: Download complete 2024-08-20T22:13:42.5806747Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:42.5807322Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:42.8236551Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:42.8237043Z 829c0e19f4b7: Download complete 2024-08-20T22:13:42.8977433Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:42.8978019Z 560cd6122a38: Download complete 2024-08-20T22:13:42.9784248Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:13:42.9784664Z 5a4eb19d7219: Download complete 2024-08-20T22:13:43.0597494Z da6a838a66c7: Verifying Checksum 2024-08-20T22:13:43.0597922Z da6a838a66c7: Download complete 2024-08-20T22:13:46.0607091Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:46.0608151Z 7ed32bc8e469: Download complete 2024-08-20T22:13:46.1420648Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:13:46.1421195Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:46.2203904Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:13:46.2204370Z 0f085d4fa279: Download complete 2024-08-20T22:13:46.7101744Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:46.7102401Z 4a0412373305: Download complete 2024-08-20T22:13:46.7840274Z 302c8c177944: Download complete 2024-08-20T22:13:46.8653762Z 02970347dff6: Download complete 2024-08-20T22:13:46.9344188Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:46.9344829Z 916c5e858016: Download complete 2024-08-20T22:13:47.0156826Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:47.0157323Z c42b3602a1fb: Download complete 2024-08-20T22:13:47.1798890Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:47.3325394Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:47.5037321Z 0c1227890755: Pull complete 2024-08-20T22:13:47.7149802Z d8d1234baab3: Pull complete 2024-08-20T22:13:53.5775889Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:53.5776512Z 6efb96fc87ea: Download complete 2024-08-20T22:13:53.6513127Z f957c1624481: Verifying Checksum 2024-08-20T22:13:53.6513735Z f957c1624481: Download complete 2024-08-20T22:13:53.7227934Z 768c31dca016: Download complete 2024-08-20T22:13:53.7904273Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:13:53.7904941Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:53.7988437Z 4f4fb700ef54: Download complete 2024-08-20T22:13:53.8734201Z 78ee77eddef2: Download complete 2024-08-20T22:13:53.9445882Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:13:53.9446452Z 37513e9b43d7: Download complete 2024-08-20T22:13:54.0628818Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:54.0629403Z a7a5392e2bf6: Download complete 2024-08-20T22:13:54.1290319Z b10de02255c0: Verifying Checksum 2024-08-20T22:13:54.1290922Z b10de02255c0: Download complete 2024-08-20T22:13:54.3740775Z e56eeff2db93: Download complete 2024-08-20T22:13:54.4564528Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:13:54.4565023Z 70ac3e38d0fd: Download complete 2024-08-20T22:13:54.5199169Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:13:54.5199800Z eb09cbf3e416: Download complete 2024-08-20T22:13:54.6812502Z ee1273153675: Verifying Checksum 2024-08-20T22:13:54.6813124Z ee1273153675: Download complete 2024-08-20T22:13:54.7626668Z e22979f9e0ee: Download complete 2024-08-20T22:13:55.3532420Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:55.3533160Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:55.4267758Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:13:55.4268266Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:22.5979689Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:22.5980178Z 5d115b29b5d5: Download complete 2024-08-20T22:14:22.6564358Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:22.6564987Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:22.7157786Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:22.7158386Z d79fd7d52698: Download complete 2024-08-20T22:14:24.2761373Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:24.2761994Z 9752d6fbbcab: Download complete 2024-08-20T22:14:24.3536646Z 83808b2eff4d: Download complete 2024-08-20T22:14:59.8565970Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:14:59.8568830Z 11fad972ef6c: Download complete 2024-08-20T22:15:02.4952759Z cbb79a705353: Verifying Checksum 2024-08-20T22:15:02.4953347Z cbb79a705353: Download complete 2024-08-20T22:15:16.8047659Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:17.0244003Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:17.2360589Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:25.5760781Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:25.7958371Z 4373aa743caf: Pull complete 2024-08-20T22:15:26.0185413Z 75b2abb16c68: Pull complete 2024-08-20T22:15:26.2399710Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:28.8580853Z 0bf61114d37b: Pull complete 2024-08-20T22:15:28.9591134Z 2df849f72ba4: Pull complete 2024-08-20T22:15:29.0762972Z 34c47d63c5de: Pull complete 2024-08-20T22:15:29.1896110Z 1fef4bd19792: Pull complete 2024-08-20T22:16:27.9696053Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:28.1898598Z b119c4fc6c58: Pull complete 2024-08-20T22:16:28.3593482Z 33b639226eee: Pull complete 2024-08-20T22:16:28.5811330Z d3547bd43146: Pull complete 2024-08-20T22:16:28.7989027Z 5645520256fa: Pull complete 2024-08-20T22:16:28.9770543Z 829ed7d91e38: Pull complete 2024-08-20T22:16:32.4946885Z 385d0421d67c: Pull complete 2024-08-20T22:16:32.7150046Z 7cbfce999d76: Pull complete 2024-08-20T22:16:32.8949445Z 6658e558ebe4: Pull complete 2024-08-20T22:16:33.0777529Z 33444a5a7e01: Pull complete 2024-08-20T22:16:33.2988946Z 7c47122f4e09: Pull complete 2024-08-20T22:16:33.5136096Z 1811ff4d3556: Pull complete 2024-08-20T22:16:44.3342511Z 7754ee068b33: Pull complete 2024-08-20T22:16:44.5096202Z 0479fbac3361: Pull complete 2024-08-20T22:16:44.6867861Z f1d6874dee9c: Pull complete 2024-08-20T22:16:45.5940213Z afc429e87b59: Pull complete 2024-08-20T22:16:45.7031805Z 019620471c25: Pull complete 2024-08-20T22:16:45.9072524Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:46.4142820Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:46.6204016Z 560cd6122a38: Pull complete 2024-08-20T22:16:46.9303317Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:47.1312937Z da6a838a66c7: Pull complete 2024-08-20T22:17:26.0116451Z cbb79a705353: Pull complete 2024-08-20T22:17:26.2069732Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:26.4332426Z 0f085d4fa279: Pull complete 2024-08-20T22:17:28.8717657Z 4a0412373305: Pull complete 2024-08-20T22:17:28.8889329Z 302c8c177944: Pull complete 2024-08-20T22:17:28.9062404Z 02970347dff6: Pull complete 2024-08-20T22:17:28.9408413Z 916c5e858016: Pull complete 2024-08-20T22:17:28.9598241Z c42b3602a1fb: Pull complete 2024-08-20T22:18:50.7451015Z 11fad972ef6c: Pull complete 2024-08-20T22:18:50.9496789Z f957c1624481: Pull complete 2024-08-20T22:18:51.1610779Z 768c31dca016: Pull complete 2024-08-20T22:18:51.5563584Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:51.7810411Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:51.9951544Z 78ee77eddef2: Pull complete 2024-08-20T22:18:52.1896849Z 37513e9b43d7: Pull complete 2024-08-20T22:18:52.5621301Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:52.7879055Z b10de02255c0: Pull complete 2024-08-20T22:18:53.0183846Z e56eeff2db93: Pull complete 2024-08-20T22:18:53.2431532Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:53.4609438Z eb09cbf3e416: Pull complete 2024-08-20T22:18:54.9130954Z ee1273153675: Pull complete 2024-08-20T22:18:55.1419704Z e22979f9e0ee: Pull complete 2024-08-20T22:18:57.3834882Z 6c1f0aabce7c: Pull complete 2024-08-20T22:18:57.6140558Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:10.9043130Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:11.1154275Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:11.3196690Z d79fd7d52698: Pull complete 2024-08-20T22:19:12.1508018Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:12.3518517Z 83808b2eff4d: Pull complete 2024-08-20T22:19:13.2510629Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:13.2980124Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:13.3219616Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:13.3322110Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:13.3323092Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:13.3334113Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:13.3334614Z env: 2024-08-20T22:19:13.3334886Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:13.3335226Z ##[endgroup] 2024-08-20T22:19:13.3534212Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:13.3534774Z with: 2024-08-20T22:19:13.3535055Z driver-version: 550.54.15 2024-08-20T22:19:13.3535385Z env: 2024-08-20T22:19:13.3535648Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:13.3535988Z ##[endgroup] 2024-08-20T22:19:13.3671112Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:13.3671674Z with: 2024-08-20T22:19:13.3671956Z timeout_minutes: 10 2024-08-20T22:19:13.3672283Z max_attempts: 3 2024-08-20T22:19:13.3705043Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:13.3739410Z retry_wait_seconds: 10 2024-08-20T22:19:13.3739778Z polling_interval_seconds: 1 2024-08-20T22:19:13.3740148Z warning_on_retry: true 2024-08-20T22:19:13.3740516Z continue_on_error: false 2024-08-20T22:19:13.3740859Z env: 2024-08-20T22:19:13.3741127Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:13.3741486Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:13.3741844Z ##[endgroup] 2024-08-20T22:19:13.4539072Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:13.4540033Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:13.4541088Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:13.7886442Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:13.7887134Z No packages marked for removal. 2024-08-20T22:19:13.7947559Z Dependencies resolved. 2024-08-20T22:19:13.7957373Z Nothing to do. 2024-08-20T22:19:13.7957783Z Complete! 2024-08-20T22:19:13.8857695Z + install_nvidia_driver_common 2024-08-20T22:19:13.8864600Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:13.8865064Z Before installing NVIDIA driver 2024-08-20T22:19:13.8865871Z + lspci 2024-08-20T22:19:13.8979845Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:13.8981350Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:13.8983204Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:13.8984808Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:13.8987098Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:13.8988165Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:13.8988934Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:13.8989802Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:13.8990446Z + lsmod 2024-08-20T22:19:13.9026632Z Module Size Used by 2024-08-20T22:19:13.9027135Z veth 36864 0 2024-08-20T22:19:13.9027570Z nvidia_modeset 1351680 0 2024-08-20T22:19:13.9028017Z video 65536 1 nvidia_modeset 2024-08-20T22:19:13.9028645Z wmi 36864 1 video 2024-08-20T22:19:13.9029046Z nvidia_uvm 4706304 0 2024-08-20T22:19:13.9029549Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:19:13.9030129Z drm 602112 1 nvidia 2024-08-20T22:19:13.9030567Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:13.9031092Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:19:13.9031612Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:13.9032277Z xt_conntrack 16384 1 2024-08-20T22:19:13.9032650Z nft_chain_nat 16384 3 2024-08-20T22:19:13.9033029Z xt_MASQUERADE 20480 1 2024-08-20T22:19:13.9033761Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:13.9034333Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:13.9034932Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:13.9035729Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:13.9036312Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:13.9036745Z xfrm_user 57344 1 2024-08-20T22:19:13.9037133Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:13.9037555Z xt_addrtype 16384 2 2024-08-20T22:19:13.9037964Z nft_compat 20480 4 2024-08-20T22:19:13.9038421Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:13.9039028Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:13.9039580Z br_netfilter 36864 0 2024-08-20T22:19:13.9039989Z bridge 307200 1 br_netfilter 2024-08-20T22:19:13.9040433Z stp 16384 1 bridge 2024-08-20T22:19:13.9040848Z llc 16384 2 bridge,stp 2024-08-20T22:19:13.9041271Z overlay 167936 0 2024-08-20T22:19:13.9041639Z tls 114688 0 2024-08-20T22:19:13.9042012Z nls_ascii 16384 1 2024-08-20T22:19:13.9042373Z nls_cp437 20480 1 2024-08-20T22:19:13.9042739Z vfat 24576 1 2024-08-20T22:19:13.9043108Z fat 86016 1 vfat 2024-08-20T22:19:13.9043503Z sunrpc 692224 1 2024-08-20T22:19:13.9043861Z ena 167936 0 2024-08-20T22:19:13.9044234Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:13.9044616Z ptp 36864 1 ena 2024-08-20T22:19:13.9045011Z pps_core 24576 1 ptp 2024-08-20T22:19:13.9045416Z i8042 45056 0 2024-08-20T22:19:13.9045784Z aesni_intel 393216 0 2024-08-20T22:19:13.9046163Z serio 28672 3 i8042 2024-08-20T22:19:13.9046636Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:13.9047125Z button 24576 0 2024-08-20T22:19:13.9047654Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:13.9048163Z sch_fq_codel 20480 17 2024-08-20T22:19:13.9048537Z dm_mod 188416 0 2024-08-20T22:19:13.9048924Z dax 45056 1 dm_mod 2024-08-20T22:19:13.9049418Z fuse 163840 1 2024-08-20T22:19:13.9049894Z loop 36864 0 2024-08-20T22:19:13.9050370Z configfs 57344 1 2024-08-20T22:19:13.9050813Z dmi_sysfs 20480 0 2024-08-20T22:19:13.9051182Z crc32_pclmul 16384 0 2024-08-20T22:19:13.9051542Z crc32c_intel 24576 0 2024-08-20T22:19:13.9051907Z efivarfs 24576 1 2024-08-20T22:19:13.9052277Z + modinfo nvidia 2024-08-20T22:19:13.9052978Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:13.9053677Z alias: char-major-195-* 2024-08-20T22:19:13.9054052Z version: 550.54.15 2024-08-20T22:19:13.9054578Z supported: external 2024-08-20T22:19:13.9054921Z license: NVIDIA 2024-08-20T22:19:13.9055288Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:13.9055769Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:13.9056227Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:13.9056690Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:13.9057182Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:13.9057673Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:13.9058168Z depends: i2c-core,drm 2024-08-20T22:19:13.9058525Z retpoline: Y 2024-08-20T22:19:13.9058827Z name: nvidia 2024-08-20T22:19:13.9059436Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:13.9060275Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:13.9060918Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:13.9061507Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:13.9061948Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:13.9062378Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:13.9062832Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:13.9063260Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:13.9063703Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:13.9064224Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:13.9064776Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:13.9065251Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:13.9065676Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:13.9066090Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:13.9066660Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:13.9067187Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:13.9067764Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:13.9068326Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:13.9068927Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:13.9069524Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:13.9070131Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:13.9070720Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:13.9071201Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:13.9071731Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:13.9072271Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:13.9072764Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:13.9073224Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:13.9073705Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:13.9074173Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:13.9074618Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:13.9075114Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:13.9075639Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:13.9076112Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:13.9076592Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:13.9077086Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:13.9077617Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:13.9078114Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:13.9078584Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:13.9078994Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:13.9079465Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:13.9079941Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:13.9080387Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:13.9080864Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:13.9081339Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:13.9081892Z parm: rm_firmware_active:charp 2024-08-20T22:19:13.9082304Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:13.9082688Z ++ command -v nvidia-smi 2024-08-20T22:19:13.9083098Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:13.9083466Z + set +e 2024-08-20T22:19:13.9083972Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:13.9349239Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:13.9349675Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:13.9350091Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:13.9350453Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:13.9350834Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:13.9351566Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:13.9352457Z + set -e 2024-08-20T22:19:13.9352774Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:13.9353353Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:13.9354050Z + post_install_nvidia_driver_common 2024-08-20T22:19:13.9356738Z + sudo modprobe nvidia 2024-08-20T22:19:14.0733279Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:14.0734023Z + lspci 2024-08-20T22:19:14.0734320Z After installing NVIDIA driver 2024-08-20T22:19:14.0841365Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:14.0842368Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:14.0843408Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:14.0844420Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:14.0845388Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:14.0846203Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:14.0846948Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:14.0847768Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:14.0848365Z + lsmod 2024-08-20T22:19:14.0875943Z Module Size Used by 2024-08-20T22:19:14.0876379Z veth 36864 0 2024-08-20T22:19:14.0876757Z nvidia_modeset 1351680 0 2024-08-20T22:19:14.0877247Z video 65536 1 nvidia_modeset 2024-08-20T22:19:14.0877827Z wmi 36864 1 video 2024-08-20T22:19:14.0878361Z nvidia_uvm 4706304 0 2024-08-20T22:19:14.0878965Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:19:14.0879614Z drm 602112 1 nvidia 2024-08-20T22:19:14.0880242Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:14.0880951Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:19:14.0881472Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:14.0881895Z xt_conntrack 16384 1 2024-08-20T22:19:14.0882275Z nft_chain_nat 16384 3 2024-08-20T22:19:14.0882649Z xt_MASQUERADE 20480 1 2024-08-20T22:19:14.0883106Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:14.0883596Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:14.0884172Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:14.0884808Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:14.0885264Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:14.0885690Z xfrm_user 57344 1 2024-08-20T22:19:14.0886084Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:14.0886497Z xt_addrtype 16384 2 2024-08-20T22:19:14.0886882Z nft_compat 20480 4 2024-08-20T22:19:14.0887338Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:14.0888105Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:14.0888648Z br_netfilter 36864 0 2024-08-20T22:19:14.0889064Z bridge 307200 1 br_netfilter 2024-08-20T22:19:14.0890659Z stp 16384 1 bridge 2024-08-20T22:19:14.0891102Z llc 16384 2 bridge,stp 2024-08-20T22:19:14.0891514Z overlay 167936 0 2024-08-20T22:19:14.0891881Z tls 114688 0 2024-08-20T22:19:14.0892247Z nls_ascii 16384 1 2024-08-20T22:19:14.0892611Z nls_cp437 20480 1 2024-08-20T22:19:14.0892966Z vfat 24576 1 2024-08-20T22:19:14.0893334Z fat 86016 1 vfat 2024-08-20T22:19:14.0893726Z sunrpc 692224 1 2024-08-20T22:19:14.0894083Z ena 167936 0 2024-08-20T22:19:14.0894463Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:14.0894851Z ptp 36864 1 ena 2024-08-20T22:19:14.0895430Z pps_core 24576 1 ptp 2024-08-20T22:19:14.0895815Z i8042 45056 0 2024-08-20T22:19:14.0896182Z aesni_intel 393216 0 2024-08-20T22:19:14.0896568Z serio 28672 3 i8042 2024-08-20T22:19:14.0897008Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:14.0897436Z button 24576 0 2024-08-20T22:19:14.0897940Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:14.0898463Z sch_fq_codel 20480 17 2024-08-20T22:19:14.0898844Z dm_mod 188416 0 2024-08-20T22:19:14.0899294Z dax 45056 1 dm_mod 2024-08-20T22:19:14.0899827Z fuse 163840 1 2024-08-20T22:19:14.0900206Z loop 36864 0 2024-08-20T22:19:14.0900568Z configfs 57344 1 2024-08-20T22:19:14.0900942Z dmi_sysfs 20480 0 2024-08-20T22:19:14.0901313Z crc32_pclmul 16384 0 2024-08-20T22:19:14.0901684Z crc32c_intel 24576 0 2024-08-20T22:19:14.0902067Z efivarfs 24576 1 2024-08-20T22:19:14.0902431Z + modinfo nvidia 2024-08-20T22:19:14.0903182Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:14.0903898Z alias: char-major-195-* 2024-08-20T22:19:14.0904290Z version: 550.54.15 2024-08-20T22:19:14.0904740Z supported: external 2024-08-20T22:19:14.0905190Z license: NVIDIA 2024-08-20T22:19:14.0905681Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:14.0906316Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:14.0906961Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:14.0907428Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:14.0907928Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:14.0908413Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:14.0908938Z depends: i2c-core,drm 2024-08-20T22:19:14.0909301Z retpoline: Y 2024-08-20T22:19:14.0909618Z name: nvidia 2024-08-20T22:19:14.0910238Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:14.0910948Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:14.0911606Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:14.0912345Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:14.0912784Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:14.0913216Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:14.0913679Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:14.0914107Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:14.0914547Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:14.0915080Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:14.0915647Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:14.0916120Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:14.0916565Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:14.0916987Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:14.0917450Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:14.0917977Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:14.0918693Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:14.0919260Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:14.0919869Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:14.0920477Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:14.0921096Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:14.0921692Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:14.0922184Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:14.0922716Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:14.0923265Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:14.0923856Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:14.0924321Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:14.0924803Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:14.0925275Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:14.0925727Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:14.0926238Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:14.0926768Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:14.0927247Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:14.0927744Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:14.0928245Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:14.0928726Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:14.0929227Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:14.0929714Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:14.0930123Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:14.0930602Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:14.0931074Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:14.0931532Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:14.0932004Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:14.0932487Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:14.0932945Z parm: rm_firmware_active:charp 2024-08-20T22:19:14.0933348Z + set +e 2024-08-20T22:19:14.0934030Z + nvidia-smi 2024-08-20T22:19:14.1111776Z Tue Aug 20 22:19:14 2024 2024-08-20T22:19:14.1112589Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:14.1113571Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:14.1114375Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:14.1115181Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:14.1116089Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:14.1116794Z | | | MIG M. | 2024-08-20T22:19:14.1117368Z |=========================================+========================+======================| 2024-08-20T22:19:14.1343159Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:14.1344132Z | 0% 25C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:19:14.1344808Z | | | N/A | 2024-08-20T22:19:14.1345684Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:14.1346572Z 2024-08-20T22:19:14.1347364Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:14.1348005Z | Processes: | 2024-08-20T22:19:14.1348913Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:14.1349791Z | ID ID Usage | 2024-08-20T22:19:14.1350531Z |=========================================================================================| 2024-08-20T22:19:14.1351235Z | No running processes found | 2024-08-20T22:19:14.1352023Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:14.4404087Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:14.4596630Z NVIDIA A10G 2024-08-20T22:19:14.4659192Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:14.4660023Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:14.4660547Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:14.4661017Z + set -e 2024-08-20T22:19:14.4661308Z INFO: Ignoring allowed status 0 2024-08-20T22:19:14.4669226Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:14.4674334Z + sudo yum install -y yum-utils 2024-08-20T22:19:14.9243362Z Last metadata expiration check: 1:37:47 ago on Tue Aug 20 20:41:27 2024. 2024-08-20T22:19:14.9472021Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:14.9777298Z Dependencies resolved. 2024-08-20T22:19:14.9902598Z Nothing to do. 2024-08-20T22:19:14.9903025Z Complete! 2024-08-20T22:19:15.0867037Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:15.0868384Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:15.0869707Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:15.3348768Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:15.4088041Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:15.9583777Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:19:15.9804230Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:16.0102653Z Dependencies resolved. 2024-08-20T22:19:16.0228287Z Nothing to do. 2024-08-20T22:19:16.0228846Z Complete! 2024-08-20T22:19:16.1164159Z + sudo systemctl restart docker 2024-08-20T22:20:16.7001040Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T22:20:16.7238527Z Tue Aug 20 22:20:16 2024 2024-08-20T22:20:16.7239239Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:16.7240068Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:16.7240907Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:16.7243874Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:16.7245206Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:16.7246148Z | | | MIG M. | 2024-08-20T22:20:16.7246870Z |=========================================+========================+======================| 2024-08-20T22:20:16.7457600Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:16.7458545Z | 0% 25C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:20:16.7459362Z | | | N/A | 2024-08-20T22:20:16.7460267Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:16.7462330Z 2024-08-20T22:20:16.7463232Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:16.7464484Z | Processes: | 2024-08-20T22:20:16.7465406Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:16.7466302Z | ID ID Usage | 2024-08-20T22:20:16.7467193Z |=========================================================================================| 2024-08-20T22:20:16.7469660Z | No running processes found | 2024-08-20T22:20:16.7470698Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:17.4935830Z Command completed after 1 attempt(s). 2024-08-20T22:20:17.5018543Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:17.5019317Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:17.5020007Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:17.5020652Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:17.5033667Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:17.5034175Z env: 2024-08-20T22:20:17.5034445Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:17.5034896Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:17.5035374Z ##[endgroup] 2024-08-20T22:20:17.7460103Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:17.7644579Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T22:20:17.7648651Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T22:20:17.8880426Z Prepare all required actions 2024-08-20T22:20:17.8880925Z Getting action download info 2024-08-20T22:20:18.0241960Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:18.2221285Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:18.3592627Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:18.3593112Z with: 2024-08-20T22:20:18.3593439Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:18.3593887Z s3-bucket: gha-artifacts 2024-08-20T22:20:18.3594219Z env: 2024-08-20T22:20:18.3594492Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:18.3594931Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:18.3595411Z ##[endgroup] 2024-08-20T22:20:18.3629293Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:18.3629733Z with: 2024-08-20T22:20:18.3630098Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:18.3630549Z s3-bucket: gha-artifacts 2024-08-20T22:20:18.3630892Z region: us-east-1 2024-08-20T22:20:18.3631188Z env: 2024-08-20T22:20:18.3631460Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:18.3631905Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:18.3632389Z ##[endgroup] 2024-08-20T22:20:18.8515684Z (node:102680) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:18.8516573Z 2024-08-20T22:20:18.8516838Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:18.8517551Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:18.8518438Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:18.9153825Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:18.9155104Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:26.4467423Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:26.4473884Z Artifact download has finished successfully 2024-08-20T22:20:26.4825352Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:26.4825798Z unzip -o artifacts.zip 2024-08-20T22:20:26.4836009Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:26.4836506Z env: 2024-08-20T22:20:26.4836773Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:26.4837220Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:26.4837701Z ##[endgroup] 2024-08-20T22:20:26.4882209Z Archive: artifacts.zip 2024-08-20T22:20:26.4883322Z creating: dist/ 2024-08-20T22:20:28.6239601Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:28.6240666Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:28.6241258Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:28.6242003Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:28.6242846Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:28.6248865Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:28.6249816Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:28.6250760Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:28.6251755Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:28.6252735Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:28.6254268Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:28.6256062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:28.6257200Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:28.6258211Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:28.6259472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:28.6261034Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:28.6262862Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:28.6264123Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:28.6265661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:28.6267162Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:28.6268402Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:28.6269426Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:28.6309062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:28.6348980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:28.6350485Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:28.6395779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:28.6397428Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:28.6399042Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:28.6401060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:28.6402932Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:28.6404841Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:28.6406706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:28.6408373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:28.6410329Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:28.6411936Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:28.6413253Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:28.6414499Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:28.6415761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:28.6416964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:28.6418205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:28.6472018Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:28.6532128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:28.6533948Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:28.6535314Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:28.6536495Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:28.6537730Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:28.6538969Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:28.6540307Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:28.6541882Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:28.6543371Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:28.6544440Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:28.6545515Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:28.6546682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:28.6547765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:28.6548843Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:28.6549907Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:28.6564500Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:28.6708056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:28.6709433Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:28.6710922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:28.6712489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:28.6714080Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:28.6715577Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:28.6716836Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:28.6718195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:28.6719346Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:28.6720487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:28.6721610Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:28.6737599Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:28.6821623Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:28.6824344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:28.6827409Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:28.6830230Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:28.6832557Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:28.6834089Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:28.6835026Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:28.6835877Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:28.6836668Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:28.6837484Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:28.6952853Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:28.7018180Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:28.7019172Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:28.7019953Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:28.7020778Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:28.7027196Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:28.7028511Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:28.7029694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:28.7030684Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:28.7031994Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:28.7033299Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:28.7034931Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:28.7035954Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:28.7037146Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:28.7038544Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:28.7039926Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:28.7041452Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:28.7042701Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:28.7044213Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:28.7045991Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:28.7047036Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:28.7048039Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:28.7086680Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:28.7126284Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:28.7127864Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:28.7173725Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:28.7175709Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:28.7177906Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:28.7179968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:28.7181767Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:28.7183669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:28.7185482Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:28.7187284Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:28.7188930Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:28.7190236Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:28.7191491Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:28.7192713Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:28.7193941Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:28.7195152Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:28.7196386Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:28.7248561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:28.7310748Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:28.7312337Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:28.7313703Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:28.7314872Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:28.7316088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:28.7317159Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:28.7318280Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:28.7319546Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:28.7321190Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:28.7322290Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:28.7323402Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:28.7324510Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:28.7325634Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:28.7326750Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:28.7327850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:28.7342924Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:28.7407315Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:28.7408683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:28.7409832Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:28.7410877Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:28.7411844Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:28.7412809Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:28.7413719Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:28.7415207Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:28.7416225Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:28.7417217Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:28.7467785Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:28.7468681Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:28.7469458Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:28.7470344Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:28.7476700Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:28.7477719Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:28.7478736Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:28.7479819Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:28.7480862Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:28.7482437Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:28.7483850Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:28.7485159Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:28.7486248Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:28.7487937Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:28.7489621Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:28.7491567Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:28.7492987Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:28.7494427Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:28.7495910Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:28.7497139Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:28.7498216Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:28.7538117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:28.7577802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:28.7579546Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:28.7624580Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:28.7626433Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:28.7628406Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:28.7630275Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:28.7631982Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:28.7633990Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:28.7635689Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:28.7637359Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:28.7638996Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:28.7640365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:28.7641671Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:28.7642958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:28.7644270Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:28.7645526Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:28.7646827Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:28.7699425Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:28.7759955Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:28.7761460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:28.7762746Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:28.7764072Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:28.7765271Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:28.7766502Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:28.7767828Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:28.7769295Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:28.7770734Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:28.7771909Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:28.7773148Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:28.7774572Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:28.7775929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:28.7777136Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:28.7778328Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:28.7779592Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:28.7902690Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:28.7904129Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:28.7905515Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:28.7907138Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:28.7908568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:28.7909856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:28.7911091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:28.7912348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:28.7913601Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:28.7914857Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:28.7916103Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:28.7930684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:28.7986402Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:28.7988168Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:28.7989539Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:28.7990774Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:28.7991929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:28.7993136Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:28.7994163Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:28.7995234Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:28.7996287Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:28.7997236Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:28.8102005Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:28.8145578Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:28.8146227Z creating: build/lib/ 2024-08-20T22:20:28.8234782Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:28.8690008Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:28.8699887Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:28.8708407Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:28.8716337Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:28.8717125Z inflating: build/lib/libclog.a 2024-08-20T22:20:28.8735776Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:28.8738329Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:28.8805452Z inflating: build/lib/libgtest.a 2024-08-20T22:20:28.8880456Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:28.8944432Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:28.8951980Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:28.8979788Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:28.9107837Z inflating: build/lib/libgloo.a 2024-08-20T22:20:28.9128460Z inflating: build/lib/libfmt.a 2024-08-20T22:20:28.9224381Z inflating: build/lib/libc10.so 2024-08-20T22:20:28.9226095Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:28.9227834Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:28.9247552Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:28.9749535Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:28.9767440Z inflating: build/lib/libgmock.a 2024-08-20T22:20:28.9768203Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:28.9769295Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:29.0159534Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:30.0376055Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:30.0950492Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:30.1007449Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:30.1008147Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:30.2302247Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:30.2558113Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:30.3065631Z inflating: build/lib/libkineto.a 2024-08-20T22:20:30.3250954Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:30.3293675Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:30.3986854Z inflating: build/lib/libonnx.a 2024-08-20T22:20:32.8556064Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:32.8560864Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:32.8565166Z inflating: build/lib/libshm.so 2024-08-20T22:20:34.9289561Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:34.9290537Z inflating: build/lib/libtorch.so 2024-08-20T22:20:34.9292818Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:35.7508248Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:35.9505653Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:35.9577747Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:35.9598933Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:35.9624968Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:35.9650844Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:35.9685458Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:35.9685934Z creating: build/bin/ 2024-08-20T22:20:35.9736846Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:35.9789296Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:35.9841834Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:35.9901763Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:35.9955190Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:36.0004696Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:36.0056339Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:36.0112811Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:36.0170747Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:36.0226887Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:36.0298154Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:36.0351434Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:36.0400481Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:36.0450799Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:36.0502193Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:36.0558870Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:36.0615988Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:36.0666235Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:36.0721506Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:36.0774092Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:36.0826423Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:36.0877117Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:36.0928538Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:36.0984244Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:36.1035061Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:36.1091557Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:36.1147712Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:36.1200689Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:36.1251462Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:36.1302153Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:36.1467115Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:36.1518208Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:36.1572983Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:36.1630210Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:36.1693202Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:36.1768185Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:36.1921153Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:36.1976366Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:36.2027603Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:36.2079920Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:36.2132289Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:36.2191022Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:36.2239995Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:36.2296235Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:36.2743587Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:36.3196736Z inflating: build/bin/protoc 2024-08-20T22:20:36.3257807Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:36.3322957Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:36.3388381Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:36.3445473Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:36.3498784Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:36.3552085Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:36.3601491Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:36.3655038Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:36.3987122Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:36.4334853Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:36.4695023Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:36.4748162Z inflating: build/bin/BackoffTest 2024-08-20T22:20:36.4801797Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:36.4858031Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:36.4911693Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:36.4925182Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:36.4979811Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:36.4984157Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:36.4987333Z inflating: build/bin/example_allreduce 2024-08-20T22:20:36.5043347Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:36.5112503Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:36.5114964Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:36.5181987Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:36.5191478Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:36.5538948Z inflating: build/bin/test_lazy 2024-08-20T22:20:36.6723547Z inflating: build/bin/test_api 2024-08-20T22:20:36.6797791Z inflating: build/bin/Dict_test 2024-08-20T22:20:36.6850322Z inflating: build/bin/Dimname_test 2024-08-20T22:20:36.6917776Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:36.6975794Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:36.7035291Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:36.7094847Z inflating: build/bin/atest 2024-08-20T22:20:36.7158002Z inflating: build/bin/basic 2024-08-20T22:20:36.7213699Z inflating: build/bin/broadcast_test 2024-08-20T22:20:36.7264312Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:36.7322596Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:36.7376212Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:36.7470484Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:36.7520649Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:36.7572882Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:36.7632478Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:36.7688639Z inflating: build/bin/half_test 2024-08-20T22:20:36.7784482Z inflating: build/bin/ivalue_test 2024-08-20T22:20:36.7835206Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:36.7890873Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:36.7945561Z inflating: build/bin/memory_format_test 2024-08-20T22:20:36.7998913Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:36.8053122Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:36.8109176Z inflating: build/bin/native_test 2024-08-20T22:20:36.8160497Z inflating: build/bin/operator_name_test 2024-08-20T22:20:36.8212711Z inflating: build/bin/operators_test 2024-08-20T22:20:36.8264461Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:36.8332332Z inflating: build/bin/pow_test 2024-08-20T22:20:36.8389809Z inflating: build/bin/quantized_test 2024-08-20T22:20:36.8440038Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:36.8491332Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:36.8548450Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:36.8608252Z inflating: build/bin/scalar_test 2024-08-20T22:20:36.8661271Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:36.8715038Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:36.8793500Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:36.8848048Z inflating: build/bin/test_parallel 2024-08-20T22:20:36.8851068Z inflating: build/bin/thread_init_test 2024-08-20T22:20:36.8907382Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:36.8968311Z inflating: build/bin/type_test 2024-08-20T22:20:36.9020195Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:36.9021879Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:36.9092113Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:36.9143413Z inflating: build/bin/weakref_test 2024-08-20T22:20:36.9195292Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:36.9247351Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:36.9307623Z inflating: build/bin/IListRef_test 2024-08-20T22:20:36.9414858Z inflating: build/bin/List_test 2024-08-20T22:20:36.9481183Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:36.9604526Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:36.9703106Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:36.9832578Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:36.9937465Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:36.9998279Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:37.0095213Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:37.0146785Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:37.0202870Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:37.0252357Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:37.0557588Z inflating: build/bin/op_registration_test 2024-08-20T22:20:37.0621697Z inflating: build/bin/inline_container_test 2024-08-20T22:20:37.0674534Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:37.0726845Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:37.0786290Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:37.0842846Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:37.0912756Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:37.0972910Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:37.1023128Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:37.1080745Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:37.1131661Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:37.1196842Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:37.1253754Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:37.1303963Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:37.1356108Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:37.1405403Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:37.1457523Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:37.1511230Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:37.1560998Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:37.1622260Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:37.1671774Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:37.1723918Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:37.1739050Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:37.1804921Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:37.1862958Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:37.1928593Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:37.1992548Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:37.2822934Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:37.3407467Z inflating: build/bin/test_jit 2024-08-20T22:20:37.3408329Z creating: .additional_ci_files/ 2024-08-20T22:20:37.3476638Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:37.3743035Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:37.3781270Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:37.3781863Z rm artifacts.zip 2024-08-20T22:20:37.3791898Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:37.3792397Z env: 2024-08-20T22:20:37.3792666Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:37.3793119Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:37.3793598Z ##[endgroup] 2024-08-20T22:20:37.4902662Z ##[group]Run df -H 2024-08-20T22:20:37.4902991Z df -H 2024-08-20T22:20:37.4912008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:37.4912503Z env: 2024-08-20T22:20:37.4912772Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:37.4913219Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:37.4913703Z ##[endgroup] 2024-08-20T22:20:37.4967724Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:37.4968301Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:37.4968927Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:37.4969567Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:37.4970022Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:37.4970709Z tmpfs 34G 8.2k 34G 1% /tmp 2024-08-20T22:20:37.4971201Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:37.4971942Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:37.5011008Z Prepare all required actions 2024-08-20T22:20:37.5011442Z Getting action download info 2024-08-20T22:20:37.6139327Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:37.6139792Z with: 2024-08-20T22:20:37.6140053Z env: 2024-08-20T22:20:37.6140330Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:37.6140777Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:37.6141272Z ##[endgroup] 2024-08-20T22:20:37.6175930Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:37.6176377Z with: 2024-08-20T22:20:37.6176643Z name: td_results 2024-08-20T22:20:37.6176954Z s3-bucket: gha-artifacts 2024-08-20T22:20:37.6177313Z region: us-east-1 2024-08-20T22:20:37.6177609Z env: 2024-08-20T22:20:37.6177901Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:37.6178387Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:37.6178870Z ##[endgroup] 2024-08-20T22:20:38.0956835Z (node:102699) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:38.0957551Z 2024-08-20T22:20:38.0957813Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:38.0958537Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:38.0959422Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:38.1529038Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:38.1535720Z Artifact download has finished successfully 2024-08-20T22:20:38.1860556Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:38.1861038Z mkdir -p .additional_ci_files 2024-08-20T22:20:38.1861609Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:38.1870811Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:38.1871297Z env: 2024-08-20T22:20:38.1871570Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.1872034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.1872514Z ##[endgroup] 2024-08-20T22:20:38.1929729Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:38.1944959Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:38.1980665Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:38.1981140Z .github/scripts/parse_ref.py 2024-08-20T22:20:38.1991013Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:38.1991364Z env: 2024-08-20T22:20:38.1991642Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.1992094Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.1992573Z ##[endgroup] 2024-08-20T22:20:38.2284044Z Prepare all required actions 2024-08-20T22:20:38.2322570Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:38.2323029Z with: 2024-08-20T22:20:38.2323573Z github-token: *** 2024-08-20T22:20:38.2323880Z env: 2024-08-20T22:20:38.2324151Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.2324606Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.2325090Z ##[endgroup] 2024-08-20T22:20:38.2343057Z ##[group]Run set -eux 2024-08-20T22:20:38.2343398Z set -eux 2024-08-20T22:20:38.2343993Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:38.2352606Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:38.2353096Z env: 2024-08-20T22:20:38.2353368Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.2353817Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.2354472Z GITHUB_TOKEN: *** 2024-08-20T22:20:38.2354775Z ##[endgroup] 2024-08-20T22:20:38.2383055Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-0035567a13af53524 2024-08-20T22:20:39.9863192Z setting job-id=29026528571 2024-08-20T22:20:39.9864088Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.0089020Z Prepare all required actions 2024-08-20T22:20:40.0089472Z Getting action download info 2024-08-20T22:20:40.1395973Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:40.1396434Z with: 2024-08-20T22:20:40.1396897Z github-token: *** 2024-08-20T22:20:40.1405873Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:40.1417299Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.1418133Z env: 2024-08-20T22:20:40.1418421Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.1418908Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.1419571Z ##[endgroup] 2024-08-20T22:20:40.1462494Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:40.1463047Z with: 2024-08-20T22:20:40.1463310Z shell: bash 2024-08-20T22:20:40.1463608Z timeout_minutes: 10 2024-08-20T22:20:40.1463925Z max_attempts: 5 2024-08-20T22:20:40.1464234Z retry_wait_seconds: 30 2024-08-20T22:20:40.1465356Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:40.1466598Z polling_interval_seconds: 1 2024-08-20T22:20:40.1466975Z warning_on_retry: true 2024-08-20T22:20:40.1467318Z continue_on_error: false 2024-08-20T22:20:40.1467653Z env: 2024-08-20T22:20:40.1467933Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.1468371Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.1469012Z GITHUB_TOKEN: *** 2024-08-20T22:20:40.1469340Z ##[endgroup] 2024-08-20T22:20:40.2232899Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:40.4611991Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:40.4792759Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T22:20:40.4796139Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T22:20:40.4909248Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T22:20:40.4918799Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T22:20:40.4927226Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:40.4932625Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:41.2019578Z Command completed after 1 attempt(s). 2024-08-20T22:20:41.2080495Z ##[group]Run set -x 2024-08-20T22:20:41.2080833Z set -x 2024-08-20T22:20:41.2081145Z  2024-08-20T22:20:41.2081688Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:41.2082368Z # in runner workspace 2024-08-20T22:20:41.2082893Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:41.2091875Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.2092378Z env: 2024-08-20T22:20:41.2092647Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.2093095Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.2093583Z ##[endgroup] 2024-08-20T22:20:41.2123937Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:41.2370494Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:41.2371042Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:41.2371504Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:41.2371913Z  2024-08-20T22:20:41.2372459Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:41.2373137Z # in runner workspace 2024-08-20T22:20:41.2373710Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:41.2374343Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:41.2374797Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:41.2384089Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:41.2393942Z  --selected-test-configs "" \ 2024-08-20T22:20:41.2394407Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:41.2394830Z  --tag "${TAG}" \ 2024-08-20T22:20:41.2395222Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:41.2395659Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:41.2396096Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:41.2405086Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.2405589Z env: 2024-08-20T22:20:41.2405859Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.2406307Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.2406986Z GITHUB_TOKEN: *** 2024-08-20T22:20:41.2407670Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.2408427Z PR_NUMBER: 2024-08-20T22:20:41.2408727Z TAG: ciflow/inductor/133712 2024-08-20T22:20:41.2409087Z EVENT_NAME: push 2024-08-20T22:20:41.2409386Z SCHEDULE: 2024-08-20T22:20:41.2409659Z HEAD_BRANCH: 2024-08-20T22:20:41.2409957Z ##[endgroup] 2024-08-20T22:20:41.2441797Z Workflow: inductor 2024-08-20T22:20:41.2442786Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.4993905Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:41.6710430Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:41.6710892Z echo "Filtered matrix:" 2024-08-20T22:20:41.6720058Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:41.6729470Z  2024-08-20T22:20:41.6729750Z echo 2024-08-20T22:20:41.6730130Z echo "Is the current job unstable? False" 2024-08-20T22:20:41.6730591Z  2024-08-20T22:20:41.6730864Z echo 2024-08-20T22:20:41.6731225Z echo "Is keep-going label set? False" 2024-08-20T22:20:41.6731668Z  2024-08-20T22:20:41.6731939Z echo 2024-08-20T22:20:41.6732248Z echo "Renabled issues? " 2024-08-20T22:20:41.6741346Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.6741834Z env: 2024-08-20T22:20:41.6742117Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.6742572Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.6743051Z ##[endgroup] 2024-08-20T22:20:41.6772489Z Filtered matrix: 2024-08-20T22:20:41.6783624Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:41.6793018Z 2024-08-20T22:20:41.6793169Z Is the current job unstable? False 2024-08-20T22:20:41.6793458Z 2024-08-20T22:20:41.6793829Z Is keep-going label set? False 2024-08-20T22:20:41.6794087Z 2024-08-20T22:20:41.6794218Z Renabled issues? 2024-08-20T22:20:41.6835772Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.6836484Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.6844801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.6845289Z env: 2024-08-20T22:20:41.6845560Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.6846020Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.6846499Z JOB_TIMEOUT: 240 2024-08-20T22:20:41.6846797Z ##[endgroup] 2024-08-20T22:20:41.6931492Z ##[group]Run set -x 2024-08-20T22:20:41.6931894Z set -x 2024-08-20T22:20:41.6932189Z  2024-08-20T22:20:41.6932544Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:41.6933088Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:41.6933950Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:41.6934467Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:41.6934877Z else 2024-08-20T22:20:41.6935216Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.6935637Z fi 2024-08-20T22:20:41.6935911Z  2024-08-20T22:20:41.6936376Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:41.6937134Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:41.6937825Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:41.6938414Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:41.6938872Z container_name=$(docker run \ 2024-08-20T22:20:41.6939294Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:41.6939667Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:41.6940069Z  -e PR_NUMBER \ 2024-08-20T22:20:41.6940474Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:41.6940865Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:41.6941268Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:41.6941654Z  -e GITHUB_JOB \ 2024-08-20T22:20:41.6942006Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:41.6942391Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:41.6942797Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:41.6943193Z  -e JOB_ID \ 2024-08-20T22:20:41.6943524Z  -e JOB_NAME \ 2024-08-20T22:20:41.6943875Z  -e BASE_SHA \ 2024-08-20T22:20:41.6944226Z  -e BRANCH \ 2024-08-20T22:20:41.6944554Z  -e SHA1 \ 2024-08-20T22:20:41.6944899Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:41.6945305Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:41.6945680Z  -e SHARD_NUMBER \ 2024-08-20T22:20:41.6946044Z  -e TEST_CONFIG \ 2024-08-20T22:20:41.6946412Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:41.6946876Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:41.6947280Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:41.6947692Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:41.6948083Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:41.6948469Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:41.6948835Z  -e NO_TD \ 2024-08-20T22:20:41.6949166Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:41.6949540Z  -e PR_LABELS \ 2024-08-20T22:20:41.6949942Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:41.6950586Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:41.6950978Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:41.6951387Z  -e XLA_CUDA \ 2024-08-20T22:20:41.6951788Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:41.6952288Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:41.6952811Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:41.6953322Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:41.6953783Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:41.6954226Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:41.6954663Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:41.6955133Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:41.6955687Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:41.6956141Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:41.6956529Z  --ipc=host \ 2024-08-20T22:20:41.6956882Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:41.6957282Z  --tty \ 2024-08-20T22:20:41.6957589Z  --detach \ 2024-08-20T22:20:41.6958093Z  --name="${container_name}" \ 2024-08-20T22:20:41.6958509Z  --user jenkins \ 2024-08-20T22:20:41.6958992Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:41.6959548Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:41.6959978Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:41.6960325Z ) 2024-08-20T22:20:41.6960723Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:41.6961636Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:41.6962608Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:41.6963517Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:41.6972645Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:41.6972988Z env: 2024-08-20T22:20:41.6973261Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.6973719Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.6974313Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:41.6974815Z PR_NUMBER: 2024-08-20T22:20:41.6975139Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:41.6975549Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:41.6975895Z GITHUB_JOB: test 2024-08-20T22:20:41.6976208Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:41.6976567Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:41.6976914Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:41.6977239Z JOB_ID: 29026528571 2024-08-20T22:20:41.6977923Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.6978671Z BRANCH: 2024-08-20T22:20:41.6979008Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.6979527Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.6980004Z TEST_CONFIG: inductor_torchbench 2024-08-20T22:20:41.6980415Z SHARD_NUMBER: 2 2024-08-20T22:20:41.6980743Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:41.6981057Z REENABLED_ISSUES: 2024-08-20T22:20:41.6981392Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:41.6981779Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:41.6982122Z TEST_SHOWLOCALS: False 2024-08-20T22:20:41.6982465Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:41.6982794Z NO_TD: False 2024-08-20T22:20:41.6983090Z TD_DISTRIBUTED: False 2024-08-20T22:20:41.6983499Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:41.6983990Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:41.6984366Z SHM_SIZE: 2g 2024-08-20T22:20:41.6985454Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:41.6986755Z XLA_CUDA: 2024-08-20T22:20:41.6987330Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:41.6987946Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:41.6988376Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:41.6988773Z DASHBOARD_TAG: 2024-08-20T22:20:41.6989274Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:41.6989651Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:41.6990023Z ##[endgroup] 2024-08-20T22:20:41.7021546Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:41.7022436Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:41.7022958Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.7030604Z +++ nproc --ignore=2 2024-08-20T22:20:41.7051078Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:52.9740038Z + container_name=6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-20T22:20:52.9744984Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:52.9746885Z + docker exec -i 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:53.1304257Z + echo DOCKER_CONTAINER_ID=6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-20T22:20:53.1308047Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:53.1311239Z + docker exec -t 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:53.5303988Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:53.8576323Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:53.8578737Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:53.8580850Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:53.8582845Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:53.8585051Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:53.8590438Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:53.8606698Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:53.8619719Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:53.8637539Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:53.9026401Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:20:54.4844125Z Installing collected packages: torch 2024-08-20T22:21:04.3629016Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:04.3631200Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:04.3632647Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:04.4454913Z + export TERM=vt100 2024-08-20T22:21:04.4455284Z + TERM=vt100 2024-08-20T22:21:04.4458136Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:04.4472700Z + source .ci/pytorch/common.sh 2024-08-20T22:21:04.4477260Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:04.4487044Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:04.4489250Z +++ declare -f -t trap_add 2024-08-20T22:21:04.4494730Z ++ set -ex 2024-08-20T22:21:04.4495216Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:04.4495722Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:04.4498840Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:04.4499598Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:04.4519189Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:04.4519675Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:04.4520099Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:04.4520494Z + shift 2024-08-20T22:21:04.4520774Z + for trap_add_name in "$@" 2024-08-20T22:21:04.4528025Z +++ trap -p EXIT 2024-08-20T22:21:04.4531332Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:04.4531819Z +++ extract_trap_cmd 2024-08-20T22:21:04.4532330Z +++ printf '%s\n' '' 2024-08-20T22:21:04.4532770Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:04.4535792Z + trap -- ' 2024-08-20T22:21:04.4536252Z cleanup_workspace' EXIT 2024-08-20T22:21:04.4536856Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:05.1697197Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:05.1721365Z + echo 'Environment variables:' 2024-08-20T22:21:05.1721790Z Environment variables: 2024-08-20T22:21:05.1722123Z + env 2024-08-20T22:21:05.1731936Z INSTALLED_DB=yes 2024-08-20T22:21:05.1732478Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:05.1733045Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:05.1733816Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:05.1734657Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:05.1735371Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:05.1735984Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:05.1736673Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:05.1737370Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:05.1737888Z HOSTNAME=6d4c93a2f182 2024-08-20T22:21:05.1738746Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.1739560Z GITHUB_ACTION=__self 2024-08-20T22:21:05.1739902Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:05.1744400Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:05.1749666Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:05.1750290Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:05.1750657Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:05.1751012Z TEST_CONFIG=inductor_torchbench 2024-08-20T22:21:05.1751463Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:05.1752000Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:05.1752500Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:05.1753014Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:05.1753431Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:05.1754096Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:05.1754634Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:05.1755146Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:05.1755688Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:05.1756017Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:05.1756462Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:05.1756858Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.1757366Z XLA_CUDA= 2024-08-20T22:21:05.1757877Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:05.1760277Z *** 2024-08-20T22:21:05.1760607Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:05.1761057Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:05.1761447Z GITHUB_ACTIONS=true 2024-08-20T22:21:05.1761819Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:05.1762406Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:05.1762998Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:05.1763542Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.1764076Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:05.1764635Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.1765431Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:05.1766216Z UCC_HOME=/usr 2024-08-20T22:21:05.1766564Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:05.1766968Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:05.1767308Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:05.1767778Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:05.1768340Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:05.1768818Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:05.1769193Z SHARD_NUMBER=2 2024-08-20T22:21:05.1769494Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:05.1769841Z HOME=/var/lib/jenkins 2024-08-20T22:21:05.1770212Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:05.1770647Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:05.1771118Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:05.1771598Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:05.1772032Z CUDA_VERSION=12.1.1 2024-08-20T22:21:05.1772464Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:05.1772894Z NUM_TEST_SHARDS=2 2024-08-20T22:21:05.1773200Z UCX_HOME=/usr 2024-08-20T22:21:05.1773718Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:05.1774801Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.1776081Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:05.1777349Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.1778630Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:05.1779378Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:05.1779706Z DASHBOARD_TAG= 2024-08-20T22:21:05.1780007Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:05.1780476Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:05.1781014Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:05.1782005Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.1782912Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:05.1783320Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:05.1783695Z PR_NUMBER= 2024-08-20T22:21:05.1783978Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:05.1784314Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:05.1784803Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:05.1785253Z TERM=vt100 2024-08-20T22:21:05.1785588Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:05.1786009Z INSTALLED_VISION=yes 2024-08-20T22:21:05.1786319Z BRANCH= 2024-08-20T22:21:05.1786594Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:05.1787308Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:05.1787748Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:05.1788511Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:05.1789265Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:05.1789749Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:05.1790216Z REENABLED_ISSUES= 2024-08-20T22:21:05.1790505Z SHLVL=1 2024-08-20T22:21:05.1790766Z MAX_JOBS=14 2024-08-20T22:21:05.1791080Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:05.1791431Z NVARCH=x86_64 2024-08-20T22:21:05.1791721Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:05.1792175Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.1792796Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:05.1793310Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:05.1793965Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:05.1794538Z GITHUB_JOB=test 2024-08-20T22:21:05.1794977Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:05.1795538Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:05.1796051Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:05.1796375Z TD_DISTRIBUTED=False 2024-08-20T22:21:05.1796752Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:05.1797183Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:05.1797601Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:05.1797964Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:05.1798309Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:05.1798652Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:05.1799700Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:05.1800774Z GITHUB_BASE_REF= 2024-08-20T22:21:05.1801094Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:05.1801454Z CI=true 2024-08-20T22:21:05.1801771Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:05.1802180Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:05.1802547Z JOB_ID=29026528571 2024-08-20T22:21:05.1802860Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:05.1803192Z GITHUB_HEAD_REF= 2024-08-20T22:21:05.1803495Z GITHUB_ACTION_REF= 2024-08-20T22:21:05.1803923Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:05.1804386Z TEST_SHOWLOCALS=False 2024-08-20T22:21:05.1804725Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:05.1805074Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:05.1805983Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.1806798Z NO_TD=False 2024-08-20T22:21:05.1807098Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:05.1807456Z _=/usr/bin/env 2024-08-20T22:21:05.1807938Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:05.1967713Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:05.1968961Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:05.1970191Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:05.1971091Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:05.1971704Z + BUILD_DIR=build 2024-08-20T22:21:05.1972034Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:05.1972428Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:05.1972764Z + SHARD_NUMBER=2 2024-08-20T22:21:05.1973069Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:05.1973452Z + export VALGRIND=ON 2024-08-20T22:21:05.1973890Z + VALGRIND=ON 2024-08-20T22:21:05.1974478Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:05.1975145Z + [[ 0 == \1 ]] 2024-08-20T22:21:05.1975449Z + [[ False == \1 ]] 2024-08-20T22:21:05.1975945Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:05.1976470Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:05.1987188Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:05.1987941Z + [[ -n '' ]] 2024-08-20T22:21:05.1988296Z + echo 'Environment variables' 2024-08-20T22:21:05.1988678Z Environment variables 2024-08-20T22:21:05.1988997Z + env 2024-08-20T22:21:05.1995988Z INSTALLED_DB=yes 2024-08-20T22:21:05.1996550Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:05.1997067Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:05.1997603Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:05.1998395Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:05.1999161Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:05.1999831Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:05.2000575Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:05.2001185Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:05.2001728Z HOSTNAME=6d4c93a2f182 2024-08-20T22:21:05.2002837Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.2003917Z GITHUB_ACTION=__self 2024-08-20T22:21:05.2004327Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:05.2008842Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:05.2013352Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:05.2013874Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:05.2014220Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:05.2014572Z TEST_CONFIG=inductor_torchbench 2024-08-20T22:21:05.2014968Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:05.2015440Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:05.2015908Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:05.2016352Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:05.2016761Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:05.2017149Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:05.2017609Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:05.2018103Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:05.2018564Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:05.2018882Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:05.2019469Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:05.2019862Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.2020314Z XLA_CUDA= 2024-08-20T22:21:05.2020775Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:05.2021161Z *** 2024-08-20T22:21:05.2021448Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:05.2021853Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:05.2022215Z GITHUB_ACTIONS=true 2024-08-20T22:21:05.2022553Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:05.2023067Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:05.2023607Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:05.2024090Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.2024610Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:05.2025100Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.2025894Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:05.2026626Z UCC_HOME=/usr 2024-08-20T22:21:05.2026968Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:05.2027540Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:05.2027989Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:05.2028476Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:05.2028984Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:05.2029491Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:05.2029888Z SHARD_NUMBER=2 2024-08-20T22:21:05.2030214Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:05.2030571Z HOME=/var/lib/jenkins 2024-08-20T22:21:05.2030932Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:05.2031390Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:05.2031876Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:05.2032360Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:05.2032724Z CUDA_VERSION=12.1.1 2024-08-20T22:21:05.2033155Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:05.2033872Z NUM_TEST_SHARDS=2 2024-08-20T22:21:05.2034182Z UCX_HOME=/usr 2024-08-20T22:21:05.2034705Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:05.2035794Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.2037064Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:05.2038319Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.2039427Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:05.2040082Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:05.2040408Z DASHBOARD_TAG= 2024-08-20T22:21:05.2040709Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:05.2041164Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:05.2041700Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:05.2042693Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.2043610Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:05.2044023Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:05.2044400Z PR_NUMBER= 2024-08-20T22:21:05.2044685Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:05.2044992Z VALGRIND=ON 2024-08-20T22:21:05.2045289Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:05.2045728Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:05.2046216Z TERM=vt100 2024-08-20T22:21:05.2046553Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:05.2046957Z INSTALLED_VISION=yes 2024-08-20T22:21:05.2047263Z BRANCH= 2024-08-20T22:21:05.2047547Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:05.2047946Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:05.2048360Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:05.2049115Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:05.2050030Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:05.2050517Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:05.2050980Z REENABLED_ISSUES= 2024-08-20T22:21:05.2051279Z SHLVL=1 2024-08-20T22:21:05.2051542Z MAX_JOBS=14 2024-08-20T22:21:05.2051855Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:05.2052209Z NVARCH=x86_64 2024-08-20T22:21:05.2052511Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:05.2052966Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:05.2053497Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:05.2053993Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:05.2054655Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:05.2055227Z GITHUB_JOB=test 2024-08-20T22:21:05.2055623Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:05.2056179Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:05.2056693Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:05.2057021Z TD_DISTRIBUTED=False 2024-08-20T22:21:05.2057399Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:05.2057964Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:05.2058394Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:05.2058751Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:05.2059098Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:05.2059447Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:05.2060488Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:05.2061547Z GITHUB_BASE_REF= 2024-08-20T22:21:05.2061869Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:05.2062234Z CI=true 2024-08-20T22:21:05.2062549Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:05.2062952Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:05.2063322Z JOB_ID=29026528571 2024-08-20T22:21:05.2063631Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:05.2063950Z GITHUB_HEAD_REF= 2024-08-20T22:21:05.2064263Z GITHUB_ACTION_REF= 2024-08-20T22:21:05.2064695Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:05.2065158Z TEST_SHOWLOCALS=False 2024-08-20T22:21:05.2065492Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:05.2065851Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:05.2066756Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_97b9d7db-a5d0-43ff-9cf4-723695957774 2024-08-20T22:21:05.2067664Z NO_TD=False 2024-08-20T22:21:05.2067960Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:05.2068321Z _=/usr/bin/env 2024-08-20T22:21:05.2068667Z + echo 'Testing pytorch' 2024-08-20T22:21:05.2069003Z Testing pytorch 2024-08-20T22:21:05.2069327Z + export LANG=C.UTF-8 2024-08-20T22:21:05.2069666Z + LANG=C.UTF-8 2024-08-20T22:21:05.2069958Z + PR_NUMBER= 2024-08-20T22:21:05.2070293Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:05.2070802Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:05.2071296Z + [[ inductor_torchbench == \s\l\o\w ]] 2024-08-20T22:21:05.2071930Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:05.2072625Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:05.2073172Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:05.2073642Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:05.2074088Z + [[ inductor_torchbench == *crossref* ]] 2024-08-20T22:21:05.2074657Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:05.2075292Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:05.2075937Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:05.2076492Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:05.2077025Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:06.2379007Z Collecting ninja==1.10.2 2024-08-20T22:21:06.2554378Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:06.3065281Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:06.9063958Z Installing collected packages: ninja 2024-08-20T22:21:06.9143001Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:06.9144311Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:06.9643889Z Successfully installed ninja-1.10.2 2024-08-20T22:21:07.0428738Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:07.0430910Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:07.0432551Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:07.0433093Z + install_tlparse 2024-08-20T22:21:07.0433757Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:07.0434745Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:07.4766233Z Collecting tlparse==0.3.25 2024-08-20T22:21:07.4937562Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:07.5475555Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:08.1937828Z Installing collected packages: tlparse 2024-08-20T22:21:08.2733736Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:08.3560333Z ++ python -m site --user-base 2024-08-20T22:21:08.3800142Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:08.3801970Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:08.3802654Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:08.3803333Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:08.3804369Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:08.3805721Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:08.3806439Z + cd test 2024-08-20T22:21:08.3806975Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:10.0144311Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:10.0144952Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:10.0147893Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:10.0148552Z + [[ inductor_torchbench == *pr_time_benchmarks* ]] 2024-08-20T22:21:10.0149249Z + [[ inductor_torchbench == *dynamo_eager* ]] 2024-08-20T22:21:10.0149864Z + [[ inductor_torchbench == *aot_eager* ]] 2024-08-20T22:21:10.0150340Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-08-20T22:21:10.0150829Z + [[ inductor_torchbench == *inductor* ]] 2024-08-20T22:21:10.0151276Z + [[ inductor_torchbench != *perf* ]] 2024-08-20T22:21:10.0151955Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:10.0152391Z + [[ inductor_torchbench == *dynamic* ]] 2024-08-20T22:21:10.0152832Z + [[ inductor_torchbench == *cpu* ]] 2024-08-20T22:21:10.0153325Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:10.0181511Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:10.0182222Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:10.0184674Z + cd test 2024-08-20T22:21:10.0185312Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:11.4858317Z PyTorch built with: 2024-08-20T22:21:11.4859070Z - GCC 9.4 2024-08-20T22:21:11.4859525Z - C++ Version: 201703 2024-08-20T22:21:11.4860427Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:11.4861924Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:11.4862598Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:11.4863135Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:11.4863632Z - NNPACK is enabled 2024-08-20T22:21:11.4864020Z - CPU capability usage: AVX2 2024-08-20T22:21:11.4864439Z - CUDA Runtime 12.1 2024-08-20T22:21:11.4864969Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:11.4865588Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:11.4866037Z - Magma 2.6.1 2024-08-20T22:21:11.4873261Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:11.4879649Z 2024-08-20T22:21:11.8086104Z + cd test 2024-08-20T22:21:11.8086977Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:13.1469381Z ATen/Parallel: 2024-08-20T22:21:13.1469905Z at::get_num_threads() : 8 2024-08-20T22:21:13.1470423Z at::get_num_interop_threads() : 16 2024-08-20T22:21:13.1470964Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:13.1471428Z omp_get_max_threads() : 8 2024-08-20T22:21:13.1472505Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:13.1473314Z mkl_get_max_threads() : 8 2024-08-20T22:21:13.1473941Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:13.1474587Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:13.1475019Z Environment variables: 2024-08-20T22:21:13.1475372Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:13.1475740Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:13.1476117Z ATen parallel backend: OpenMP 2024-08-20T22:21:13.1476395Z 2024-08-20T22:21:13.4278812Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:13.4279598Z + [[ inductor_torchbench == *backward* ]] 2024-08-20T22:21:13.4280050Z + [[ inductor_torchbench == *xla* ]] 2024-08-20T22:21:13.4280495Z + [[ inductor_torchbench == *executorch* ]] 2024-08-20T22:21:13.4280981Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:13.4281653Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:13.4282373Z + [[ inductor_torchbench == distributed ]] 2024-08-20T22:21:13.4283024Z + [[ inductor_torchbench == *inductor_distributed* ]] 2024-08-20T22:21:13.4283629Z + [[ inductor_torchbench == *inductor-halide* ]] 2024-08-20T22:21:13.4284235Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]] 2024-08-20T22:21:13.4284764Z + [[ inductor_torchbench == *huggingface* ]] 2024-08-20T22:21:13.4285222Z + [[ inductor_torchbench == *timm* ]] 2024-08-20T22:21:13.4285657Z + [[ inductor_torchbench == *torchbench* ]] 2024-08-20T22:21:13.4286505Z + [[ inductor_torchbench == *cpu* ]] 2024-08-20T22:21:13.4286914Z + install_torchaudio cuda 2024-08-20T22:21:13.4287255Z + local commit 2024-08-20T22:21:13.4287569Z ++ get_pinned_commit audio 2024-08-20T22:21:13.4287952Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T22:21:13.4300831Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:13.4301381Z + [[ cuda == \c\u\d\a ]] 2024-08-20T22:21:13.4301821Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:21:13.4302750Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:13.4304215Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:13.7656685Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:13.7660591Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-b0q3w4rr 2024-08-20T22:21:13.7691714Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-b0q3w4rr 2024-08-20T22:21:14.5651845Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T22:21:14.5679931Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:14.9679425Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:14.9680584Z Running command git submodule update --init --recursive -q 2024-08-20T22:21:17.3873764Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:21:17.3903176Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:17.3930253Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T22:21:17.3934617Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T22:21:17.3938145Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T22:21:17.3941454Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T22:21:17.3944603Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T22:21:17.3949989Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1) 2024-08-20T22:21:17.3964712Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T22:21:17.4445676Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T22:21:17.4508468Z Building wheels for collected packages: torchaudio 2024-08-20T22:23:50.1775411Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:23:50.1810872Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310394 sha256=bb538d5cde76c912940964f481bb96d42e71bfc164975425621b739ebd4912f0 2024-08-20T22:23:50.1812927Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95 2024-08-20T22:23:50.1842983Z Successfully built torchaudio 2024-08-20T22:23:50.6703877Z Installing collected packages: torchaudio 2024-08-20T22:23:50.8854162Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T22:23:51.0675407Z + install_torchtext 2024-08-20T22:23:51.0675797Z + local data_commit 2024-08-20T22:23:51.0676126Z + local text_commit 2024-08-20T22:23:51.0678603Z ++ get_pinned_commit data 2024-08-20T22:23:51.0679001Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T22:23:51.0704947Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.0709732Z ++ get_pinned_commit text 2024-08-20T22:23:51.0710119Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T22:23:51.0728071Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:23:51.0729767Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.0731252Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.4112933Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.4117988Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-up3wkpz_ 2024-08-20T22:23:51.4169541Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-up3wkpz_ 2024-08-20T22:23:51.8626827Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T22:23:51.8655603Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.1212354Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.2878119Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.2879821Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:15.6610648Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:15.6648626Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T22:25:15.6651421Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T22:25:15.6655648Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:15.6685375Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T22:25:15.6689531Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T22:25:15.6693212Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T22:25:15.6696597Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T22:25:15.6699881Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T22:25:15.6705033Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1) 2024-08-20T22:25:15.6720319Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T22:25:15.6783261Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T22:25:15.6785947Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T22:25:15.6791054Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T22:25:15.7285783Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T22:25:15.7349577Z Building wheels for collected packages: torchdata 2024-08-20T22:25:17.7007179Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T22:25:17.7015181Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=b02d3e7a19142421507cf3d02a8f38835d5fe9c6f6a5bcf0cd890396ae3c0e77 2024-08-20T22:25:17.7016953Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-08-20T22:25:17.7043934Z Successfully built torchdata 2024-08-20T22:25:18.1800563Z Installing collected packages: torchdata 2024-08-20T22:25:18.2809505Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T22:25:23.3248198Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:23.3249856Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:23.6630641Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:23.6633976Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-fz51c5am 2024-08-20T22:25:23.6661606Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-fz51c5am 2024-08-20T22:25:24.4891623Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T22:25:24.4920668Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:24.9826732Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:25.1501901Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:25.1503104Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:32.5783271Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:32.5822920Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T22:25:32.5826429Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T22:25:32.5829314Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:32.5832811Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4) 2024-08-20T22:25:32.5836677Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T22:25:32.5863014Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T22:25:32.5866892Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T22:25:32.5871179Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T22:25:32.5874763Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T22:25:32.5901208Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T22:25:32.5904962Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T22:25:32.5907647Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T22:25:32.5911771Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T22:25:32.5914212Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T22:25:32.5919490Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1) 2024-08-20T22:25:32.5934141Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T22:25:32.6511495Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T22:25:32.6577543Z Building wheels for collected packages: torchtext 2024-08-20T22:26:11.4795603Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:26:11.4827893Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045465 sha256=51d1d3d864c9893f970cb6e64293575ef3d666a7a335d6947e2aff6766bf5b6a 2024-08-20T22:26:11.4829544Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-08-20T22:26:11.4855682Z Successfully built torchtext 2024-08-20T22:26:11.9622809Z Installing collected packages: torchtext 2024-08-20T22:26:12.0745214Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T22:26:12.2338030Z + install_torchvision 2024-08-20T22:26:12.2338398Z + local orig_preload 2024-08-20T22:26:12.2339047Z + local commit 2024-08-20T22:26:12.2342012Z ++ get_pinned_commit vision 2024-08-20T22:26:12.2342449Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:26:12.2359879Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:12.2360494Z + orig_preload= 2024-08-20T22:26:12.2360988Z + '[' -n '' ']' 2024-08-20T22:26:12.2361956Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:12.2363425Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:12.5727302Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:12.5732070Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-0pj63dkx 2024-08-20T22:26:12.5762984Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-0pj63dkx 2024-08-20T22:26:14.1068565Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:26:14.1098470Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:15.4374554Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:15.7236776Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:18.2191904Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:26:18.2226523Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:26:18.2231405Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:26:18.2234722Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:26:18.2305025Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:26:18.2309343Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:26:18.2312885Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:26:18.2316148Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:26:18.2319829Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:26:18.2324977Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:26:18.2341055Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:26:18.2825596Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:26:18.2888305Z Building wheels for collected packages: torchvision 2024-08-20T22:27:37.6602600Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:27:37.6636124Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2115831 sha256=b2329edefa1a568818b19a3a84cc0791e2fb170979b46fff7c2500c1084edacb 2024-08-20T22:27:37.6639417Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:27:37.6671700Z Successfully built torchvision 2024-08-20T22:27:38.1493672Z Installing collected packages: torchvision 2024-08-20T22:27:38.5659925Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:27:38.7114519Z + '[' -n '' ']' 2024-08-20T22:27:38.7115099Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:27:38.7115609Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T22:27:38.7116352Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T22:27:39.0492378Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T22:27:39.0496950Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-3dop1mxw 2024-08-20T22:27:39.0527451Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-3dop1mxw 2024-08-20T22:27:39.6327948Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:27:42.0801100Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:27:42.0836636Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T22:29:01.3056017Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - done 2024-08-20T22:29:01.3074604Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928013 sha256=481fb410b9887ddd77679a2c5759a7ae534e4fe911095989ca3c71819d91de3d 2024-08-20T22:29:01.3076425Z Stored in directory: /tmp/pip-ephem-wheel-cache-0w5s2zbh/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2024-08-20T22:29:01.3105822Z Successfully built torchao 2024-08-20T22:29:01.8552204Z Installing collected packages: torchao 2024-08-20T22:29:02.0975184Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T22:29:02.2067679Z + id=1 2024-08-20T22:29:02.2068944Z + pip_install opencv-python==4.8.0.74 2024-08-20T22:29:02.2069561Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T22:29:02.6697201Z Collecting opencv-python==4.8.0.74 2024-08-20T22:29:02.6875006Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T22:29:02.6957750Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2024-08-20T22:29:02.7030127Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T22:29:03.9912661Z Installing collected packages: opencv-python 2024-08-20T22:29:04.8591049Z Successfully installed opencv-python-4.8.0.74 2024-08-20T22:29:04.9583886Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T22:29:04.9584617Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T22:29:04.9587113Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-08-20T22:29:04.9587780Z + checkout_install_torchbench 2024-08-20T22:29:04.9588448Z + local commit 2024-08-20T22:29:04.9589978Z ++ get_pinned_commit torchbench 2024-08-20T22:29:04.9590425Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T22:29:04.9607875Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:04.9608487Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T22:29:04.9624135Z Cloning into 'torchbench'... 2024-08-20T22:29:05.1191776Z remote: Enumerating objects: 31006, done. 2024-08-20T22:29:05.1196534Z remote: Counting objects: 0% (1/1746) 2024-08-20T22:29:05.1197687Z remote: Counting objects: 1% (18/1746) 2024-08-20T22:29:05.1200715Z remote: Counting objects: 2% (35/1746) 2024-08-20T22:29:05.1201461Z remote: Counting objects: 3% (53/1746) 2024-08-20T22:29:05.1202180Z remote: Counting objects: 4% (70/1746) 2024-08-20T22:29:05.1202815Z remote: Counting objects: 5% (88/1746) 2024-08-20T22:29:05.1203599Z remote: Counting objects: 6% (105/1746) 2024-08-20T22:29:05.1204319Z remote: Counting objects: 7% (123/1746) 2024-08-20T22:29:05.1204907Z remote: Counting objects: 8% (140/1746) 2024-08-20T22:29:05.1205649Z remote: Counting objects: 9% (158/1746) 2024-08-20T22:29:05.1206396Z remote: Counting objects: 10% (175/1746) 2024-08-20T22:29:05.1207015Z remote: Counting objects: 11% (193/1746) 2024-08-20T22:29:05.1207698Z remote: Counting objects: 12% (210/1746) 2024-08-20T22:29:05.1208229Z remote: Counting objects: 13% (227/1746) 2024-08-20T22:29:05.1208746Z remote: Counting objects: 14% (245/1746) 2024-08-20T22:29:05.1209258Z remote: Counting objects: 15% (262/1746) 2024-08-20T22:29:05.1209944Z remote: Counting objects: 16% (280/1746) 2024-08-20T22:29:05.1210714Z remote: Counting objects: 17% (297/1746) 2024-08-20T22:29:05.1212689Z remote: Counting objects: 18% (315/1746) 2024-08-20T22:29:05.1213213Z remote: Counting objects: 19% (332/1746) 2024-08-20T22:29:05.1213743Z remote: Counting objects: 20% (350/1746) 2024-08-20T22:29:05.1214400Z remote: Counting objects: 21% (367/1746) 2024-08-20T22:29:05.1215036Z remote: Counting objects: 22% (385/1746) 2024-08-20T22:29:05.1216392Z remote: Counting objects: 23% (402/1746) 2024-08-20T22:29:05.1217132Z remote: Counting objects: 24% (420/1746) 2024-08-20T22:29:05.1217838Z remote: Counting objects: 25% (437/1746) 2024-08-20T22:29:05.1218417Z remote: Counting objects: 26% (454/1746) 2024-08-20T22:29:05.1218956Z remote: Counting objects: 27% (472/1746) 2024-08-20T22:29:05.1219481Z remote: Counting objects: 28% (489/1746) 2024-08-20T22:29:05.1219996Z remote: Counting objects: 29% (507/1746) 2024-08-20T22:29:05.1220519Z remote: Counting objects: 30% (524/1746) 2024-08-20T22:29:05.1221054Z remote: Counting objects: 31% (542/1746) 2024-08-20T22:29:05.1221569Z remote: Counting objects: 32% (559/1746) 2024-08-20T22:29:05.1222354Z remote: Counting objects: 33% (577/1746) 2024-08-20T22:29:05.1223091Z remote: Counting objects: 34% (594/1746) 2024-08-20T22:29:05.1223741Z remote: Counting objects: 35% (612/1746) 2024-08-20T22:29:05.1224460Z remote: Counting objects: 36% (629/1746) 2024-08-20T22:29:05.1225165Z remote: Counting objects: 37% (647/1746) 2024-08-20T22:29:05.1225868Z remote: Counting objects: 38% (664/1746) 2024-08-20T22:29:05.1226453Z remote: Counting objects: 39% (681/1746) 2024-08-20T22:29:05.1226973Z remote: Counting objects: 40% (699/1746) 2024-08-20T22:29:05.1227491Z remote: Counting objects: 41% (716/1746) 2024-08-20T22:29:05.1228225Z remote: Counting objects: 42% (734/1746) 2024-08-20T22:29:05.1228758Z remote: Counting objects: 43% (751/1746) 2024-08-20T22:29:05.1229282Z remote: Counting objects: 44% (769/1746) 2024-08-20T22:29:05.1229820Z remote: Counting objects: 45% (786/1746) 2024-08-20T22:29:05.1230331Z remote: Counting objects: 46% (804/1746) 2024-08-20T22:29:05.1230853Z remote: Counting objects: 47% (821/1746) 2024-08-20T22:29:05.1231372Z remote: Counting objects: 48% (839/1746) 2024-08-20T22:29:05.1231887Z remote: Counting objects: 49% (856/1746) 2024-08-20T22:29:05.1232402Z remote: Counting objects: 50% (873/1746) 2024-08-20T22:29:05.1232919Z remote: Counting objects: 51% (891/1746) 2024-08-20T22:29:05.1233787Z remote: Counting objects: 52% (908/1746) 2024-08-20T22:29:05.1234328Z remote: Counting objects: 53% (926/1746) 2024-08-20T22:29:05.1234853Z remote: Counting objects: 54% (943/1746) 2024-08-20T22:29:05.1235369Z remote: Counting objects: 55% (961/1746) 2024-08-20T22:29:05.1235906Z remote: Counting objects: 56% (978/1746) 2024-08-20T22:29:05.1236422Z remote: Counting objects: 57% (996/1746) 2024-08-20T22:29:05.1236943Z remote: Counting objects: 58% (1013/1746) 2024-08-20T22:29:05.1237486Z remote: Counting objects: 59% (1031/1746) 2024-08-20T22:29:05.1238018Z remote: Counting objects: 60% (1048/1746) 2024-08-20T22:29:05.1238557Z remote: Counting objects: 61% (1066/1746) 2024-08-20T22:29:05.1239079Z remote: Counting objects: 62% (1083/1746) 2024-08-20T22:29:05.1239604Z remote: Counting objects: 63% (1100/1746) 2024-08-20T22:29:05.1240132Z remote: Counting objects: 64% (1118/1746) 2024-08-20T22:29:05.1240660Z remote: Counting objects: 65% (1135/1746) 2024-08-20T22:29:05.1241183Z remote: Counting objects: 66% (1153/1746) 2024-08-20T22:29:05.1241708Z remote: Counting objects: 67% (1170/1746) 2024-08-20T22:29:05.1242236Z remote: Counting objects: 68% (1188/1746) 2024-08-20T22:29:05.1242754Z remote: Counting objects: 69% (1205/1746) 2024-08-20T22:29:05.1243279Z remote: Counting objects: 70% (1223/1746) 2024-08-20T22:29:05.1243800Z remote: Counting objects: 71% (1240/1746) 2024-08-20T22:29:05.1244327Z remote: Counting objects: 72% (1258/1746) 2024-08-20T22:29:05.1245027Z remote: Counting objects: 73% (1275/1746) 2024-08-20T22:29:05.1245562Z remote: Counting objects: 74% (1293/1746) 2024-08-20T22:29:05.1246085Z remote: Counting objects: 75% (1310/1746) 2024-08-20T22:29:05.1246605Z remote: Counting objects: 76% (1327/1746) 2024-08-20T22:29:05.1247128Z remote: Counting objects: 77% (1345/1746) 2024-08-20T22:29:05.1247653Z remote: Counting objects: 78% (1362/1746) 2024-08-20T22:29:05.1248174Z remote: Counting objects: 79% (1380/1746) 2024-08-20T22:29:05.1248696Z remote: Counting objects: 80% (1397/1746) 2024-08-20T22:29:05.1249215Z remote: Counting objects: 81% (1415/1746) 2024-08-20T22:29:05.1249738Z remote: Counting objects: 82% (1432/1746) 2024-08-20T22:29:05.1250265Z remote: Counting objects: 83% (1450/1746) 2024-08-20T22:29:05.1250787Z remote: Counting objects: 84% (1467/1746) 2024-08-20T22:29:05.1251305Z remote: Counting objects: 85% (1485/1746) 2024-08-20T22:29:05.1251841Z remote: Counting objects: 86% (1502/1746) 2024-08-20T22:29:05.1252368Z remote: Counting objects: 87% (1520/1746) 2024-08-20T22:29:05.1253007Z remote: Counting objects: 88% (1537/1746) 2024-08-20T22:29:05.1253541Z remote: Counting objects: 89% (1554/1746) 2024-08-20T22:29:05.1254063Z remote: Counting objects: 90% (1572/1746) 2024-08-20T22:29:05.1254588Z remote: Counting objects: 91% (1589/1746) 2024-08-20T22:29:05.1255104Z remote: Counting objects: 92% (1607/1746) 2024-08-20T22:29:05.1255631Z remote: Counting objects: 93% (1624/1746) 2024-08-20T22:29:05.1256160Z remote: Counting objects: 94% (1642/1746) 2024-08-20T22:29:05.1256682Z remote: Counting objects: 95% (1659/1746) 2024-08-20T22:29:05.1257202Z remote: Counting objects: 96% (1677/1746) 2024-08-20T22:29:05.1257726Z remote: Counting objects: 97% (1694/1746) 2024-08-20T22:29:05.1258248Z remote: Counting objects: 98% (1712/1746) 2024-08-20T22:29:05.1258772Z remote: Counting objects: 99% (1729/1746) 2024-08-20T22:29:05.1259297Z remote: Counting objects: 100% (1746/1746) 2024-08-20T22:29:05.1261319Z remote: Counting objects: 100% (1746/1746), done. 2024-08-20T22:29:05.1261894Z remote: Compressing objects: 0% (1/902) 2024-08-20T22:29:05.1316506Z remote: Compressing objects: 1% (10/902) 2024-08-20T22:29:05.1438497Z remote: Compressing objects: 2% (19/902) 2024-08-20T22:29:05.1525860Z remote: Compressing objects: 3% (28/902) 2024-08-20T22:29:05.1577166Z remote: Compressing objects: 4% (37/902) 2024-08-20T22:29:05.1629424Z remote: Compressing objects: 5% (46/902) 2024-08-20T22:29:05.1661222Z remote: Compressing objects: 6% (55/902) 2024-08-20T22:29:05.1688176Z remote: Compressing objects: 7% (64/902) 2024-08-20T22:29:05.1727453Z remote: Compressing objects: 8% (73/902) 2024-08-20T22:29:05.1761070Z remote: Compressing objects: 9% (82/902) 2024-08-20T22:29:05.1789600Z remote: Compressing objects: 10% (91/902) 2024-08-20T22:29:05.1803165Z remote: Compressing objects: 11% (100/902) 2024-08-20T22:29:05.1816115Z remote: Compressing objects: 12% (109/902) 2024-08-20T22:29:05.1831356Z remote: Compressing objects: 13% (118/902) 2024-08-20T22:29:05.1847373Z remote: Compressing objects: 14% (127/902) 2024-08-20T22:29:05.1856219Z remote: Compressing objects: 15% (136/902) 2024-08-20T22:29:05.1865165Z remote: Compressing objects: 16% (145/902) 2024-08-20T22:29:05.1872320Z remote: Compressing objects: 17% (154/902) 2024-08-20T22:29:05.1875404Z remote: Compressing objects: 18% (163/902) 2024-08-20T22:29:05.1878948Z remote: Compressing objects: 19% (172/902) 2024-08-20T22:29:05.1880880Z remote: Compressing objects: 20% (181/902) 2024-08-20T22:29:05.1881635Z remote: Compressing objects: 21% (190/902) 2024-08-20T22:29:05.1882251Z remote: Compressing objects: 22% (199/902) 2024-08-20T22:29:05.1886722Z remote: Compressing objects: 23% (208/902) 2024-08-20T22:29:05.1887295Z remote: Compressing objects: 24% (217/902) 2024-08-20T22:29:05.1888399Z remote: Compressing objects: 25% (226/902) 2024-08-20T22:29:05.1891272Z remote: Compressing objects: 26% (235/902) 2024-08-20T22:29:05.1899527Z remote: Compressing objects: 27% (244/902) 2024-08-20T22:29:05.1902201Z remote: Compressing objects: 28% (253/902) 2024-08-20T22:29:05.1906256Z remote: Compressing objects: 29% (262/902) 2024-08-20T22:29:05.1909519Z remote: Compressing objects: 30% (271/902) 2024-08-20T22:29:05.1911985Z remote: Compressing objects: 31% (280/902) 2024-08-20T22:29:05.1915524Z remote: Compressing objects: 32% (289/902) 2024-08-20T22:29:05.1918807Z remote: Compressing objects: 33% (298/902) 2024-08-20T22:29:05.1923134Z remote: Compressing objects: 34% (307/902) 2024-08-20T22:29:05.1925115Z remote: Compressing objects: 35% (316/902) 2024-08-20T22:29:05.1927750Z remote: Compressing objects: 36% (325/902) 2024-08-20T22:29:05.1930526Z remote: Compressing objects: 37% (334/902) 2024-08-20T22:29:05.1932820Z remote: Compressing objects: 38% (343/902) 2024-08-20T22:29:05.1935089Z remote: Compressing objects: 39% (352/902) 2024-08-20T22:29:05.1936508Z remote: Compressing objects: 40% (361/902) 2024-08-20T22:29:05.1939166Z remote: Compressing objects: 41% (370/902) 2024-08-20T22:29:05.1939839Z remote: Compressing objects: 42% (379/902) 2024-08-20T22:29:05.1940430Z remote: Compressing objects: 43% (388/902) 2024-08-20T22:29:05.1941047Z remote: Compressing objects: 44% (397/902) 2024-08-20T22:29:05.1942289Z remote: Compressing objects: 45% (406/902) 2024-08-20T22:29:05.1942931Z remote: Compressing objects: 46% (415/902) 2024-08-20T22:29:05.1943573Z remote: Compressing objects: 47% (424/902) 2024-08-20T22:29:05.1944245Z remote: Compressing objects: 48% (433/902) 2024-08-20T22:29:05.1944912Z remote: Compressing objects: 49% (442/902) 2024-08-20T22:29:05.1945516Z remote: Compressing objects: 50% (451/902) 2024-08-20T22:29:05.1946247Z remote: Compressing objects: 51% (461/902) 2024-08-20T22:29:05.1946987Z remote: Compressing objects: 52% (470/902) 2024-08-20T22:29:05.1947700Z remote: Compressing objects: 53% (479/902) 2024-08-20T22:29:05.1948502Z remote: Compressing objects: 54% (488/902) 2024-08-20T22:29:05.1949152Z remote: Compressing objects: 55% (497/902) 2024-08-20T22:29:05.1949692Z remote: Compressing objects: 56% (506/902) 2024-08-20T22:29:05.1950327Z remote: Compressing objects: 57% (515/902) 2024-08-20T22:29:05.1950881Z remote: Compressing objects: 58% (524/902) 2024-08-20T22:29:05.1951427Z remote: Compressing objects: 59% (533/902) 2024-08-20T22:29:05.1951964Z remote: Compressing objects: 60% (542/902) 2024-08-20T22:29:05.1952513Z remote: Compressing objects: 61% (551/902) 2024-08-20T22:29:05.1953058Z remote: Compressing objects: 62% (560/902) 2024-08-20T22:29:05.1953601Z remote: Compressing objects: 63% (569/902) 2024-08-20T22:29:05.1954143Z remote: Compressing objects: 64% (578/902) 2024-08-20T22:29:05.1954686Z remote: Compressing objects: 65% (587/902) 2024-08-20T22:29:05.1955237Z remote: Compressing objects: 66% (596/902) 2024-08-20T22:29:05.1955894Z remote: Compressing objects: 67% (605/902) 2024-08-20T22:29:05.1956437Z remote: Compressing objects: 68% (614/902) 2024-08-20T22:29:05.1956972Z remote: Compressing objects: 69% (623/902) 2024-08-20T22:29:05.1957509Z remote: Compressing objects: 70% (632/902) 2024-08-20T22:29:05.1958044Z remote: Compressing objects: 71% (641/902) 2024-08-20T22:29:05.1958572Z remote: Compressing objects: 72% (650/902) 2024-08-20T22:29:05.1959110Z remote: Compressing objects: 73% (659/902) 2024-08-20T22:29:05.1959647Z remote: Compressing objects: 74% (668/902) 2024-08-20T22:29:05.1960198Z remote: Compressing objects: 75% (677/902) 2024-08-20T22:29:05.1967282Z remote: Compressing objects: 76% (686/902) 2024-08-20T22:29:05.1975011Z remote: Compressing objects: 77% (695/902) 2024-08-20T22:29:05.1980541Z remote: Compressing objects: 78% (704/902) 2024-08-20T22:29:05.1986293Z remote: Compressing objects: 79% (713/902) 2024-08-20T22:29:05.1992867Z remote: Compressing objects: 80% (722/902) 2024-08-20T22:29:05.2006218Z remote: Compressing objects: 81% (731/902) 2024-08-20T22:29:05.2015194Z remote: Compressing objects: 82% (740/902) 2024-08-20T22:29:05.2016276Z remote: Compressing objects: 83% (749/902) 2024-08-20T22:29:05.2016828Z remote: Compressing objects: 84% (758/902) 2024-08-20T22:29:05.2020497Z remote: Compressing objects: 85% (767/902) 2024-08-20T22:29:05.2025521Z remote: Compressing objects: 86% (776/902) 2024-08-20T22:29:05.2031865Z remote: Compressing objects: 87% (785/902) 2024-08-20T22:29:05.2036167Z remote: Compressing objects: 88% (794/902) 2024-08-20T22:29:05.2038163Z remote: Compressing objects: 89% (803/902) 2024-08-20T22:29:05.2042068Z remote: Compressing objects: 90% (812/902) 2024-08-20T22:29:05.2045364Z remote: Compressing objects: 91% (821/902) 2024-08-20T22:29:05.2048373Z remote: Compressing objects: 92% (830/902) 2024-08-20T22:29:05.2053150Z remote: Compressing objects: 93% (839/902) 2024-08-20T22:29:05.2053866Z remote: Compressing objects: 94% (848/902) 2024-08-20T22:29:05.2057860Z remote: Compressing objects: 95% (857/902) 2024-08-20T22:29:05.2059677Z remote: Compressing objects: 96% (866/902) 2024-08-20T22:29:05.2062808Z remote: Compressing objects: 97% (875/902) 2024-08-20T22:29:05.2063668Z remote: Compressing objects: 98% (884/902) 2024-08-20T22:29:05.2064685Z remote: Compressing objects: 99% (893/902) 2024-08-20T22:29:05.2065240Z remote: Compressing objects: 100% (902/902) 2024-08-20T22:29:05.2065819Z remote: Compressing objects: 100% (902/902), done. 2024-08-20T22:29:05.2147703Z Receiving objects: 0% (1/31006) 2024-08-20T22:29:05.2189352Z Receiving objects: 1% (311/31006) 2024-08-20T22:29:05.2228989Z Receiving objects: 2% (621/31006) 2024-08-20T22:29:05.2265489Z Receiving objects: 3% (931/31006) 2024-08-20T22:29:05.2298755Z Receiving objects: 4% (1241/31006) 2024-08-20T22:29:05.2331753Z Receiving objects: 5% (1551/31006) 2024-08-20T22:29:05.2369676Z Receiving objects: 6% (1861/31006) 2024-08-20T22:29:05.2409002Z Receiving objects: 7% (2171/31006) 2024-08-20T22:29:05.2439776Z Receiving objects: 8% (2481/31006) 2024-08-20T22:29:05.2472782Z Receiving objects: 9% (2791/31006) 2024-08-20T22:29:05.2543281Z Receiving objects: 10% (3101/31006) 2024-08-20T22:29:05.2591789Z Receiving objects: 11% (3411/31006) 2024-08-20T22:29:05.2659712Z Receiving objects: 12% (3721/31006) 2024-08-20T22:29:05.2792752Z Receiving objects: 13% (4031/31006) 2024-08-20T22:29:05.2936944Z Receiving objects: 14% (4341/31006) 2024-08-20T22:29:05.3008647Z Receiving objects: 15% (4651/31006) 2024-08-20T22:29:05.3082178Z Receiving objects: 16% (4961/31006) 2024-08-20T22:29:05.3141852Z Receiving objects: 17% (5272/31006) 2024-08-20T22:29:05.3170661Z Receiving objects: 18% (5582/31006) 2024-08-20T22:29:05.3228556Z Receiving objects: 19% (5892/31006) 2024-08-20T22:29:05.3262580Z Receiving objects: 20% (6202/31006) 2024-08-20T22:29:05.3278164Z Receiving objects: 21% (6512/31006) 2024-08-20T22:29:05.3461149Z Receiving objects: 22% (6822/31006) 2024-08-20T22:29:06.2094534Z Receiving objects: 23% (7132/31006) 2024-08-20T22:29:06.3266369Z Receiving objects: 23% (7377/31006), 75.69 MiB | 75.68 MiB/s 2024-08-20T22:29:06.8765695Z Receiving objects: 24% (7442/31006), 75.69 MiB | 75.68 MiB/s 2024-08-20T22:29:06.9488305Z Receiving objects: 25% (7752/31006), 113.35 MiB | 75.56 MiB/s 2024-08-20T22:29:07.0212606Z Receiving objects: 26% (8062/31006), 113.35 MiB | 75.56 MiB/s 2024-08-20T22:29:07.0925533Z Receiving objects: 27% (8372/31006), 113.35 MiB | 75.56 MiB/s 2024-08-20T22:29:07.1638037Z Receiving objects: 28% (8682/31006), 113.35 MiB | 75.56 MiB/s 2024-08-20T22:29:07.2093663Z Receiving objects: 29% (8992/31006), 113.35 MiB | 75.56 MiB/s 2024-08-20T22:29:07.2361692Z Receiving objects: 29% (9188/31006), 151.67 MiB | 75.87 MiB/s 2024-08-20T22:29:07.3080182Z Receiving objects: 30% (9302/31006), 151.67 MiB | 75.87 MiB/s 2024-08-20T22:29:07.3799446Z Receiving objects: 31% (9612/31006), 151.67 MiB | 75.87 MiB/s 2024-08-20T22:29:07.6136524Z Receiving objects: 32% (9922/31006), 151.67 MiB | 75.87 MiB/s 2024-08-20T22:29:07.6877973Z Receiving objects: 33% (10232/31006), 151.67 MiB | 75.87 MiB/s 2024-08-20T22:29:07.7001449Z Receiving objects: 34% (10543/31006), 151.67 MiB | 75.87 MiB/s 2024-08-20T22:29:07.9600001Z Receiving objects: 35% (10853/31006), 151.67 MiB | 75.87 MiB/s 2024-08-20T22:29:08.2099091Z Receiving objects: 36% (11163/31006), 184.89 MiB | 73.98 MiB/s 2024-08-20T22:29:08.2119548Z Receiving objects: 36% (11450/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2138655Z Receiving objects: 37% (11473/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2161501Z Receiving objects: 38% (11783/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2180592Z Receiving objects: 39% (12093/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2196165Z Receiving objects: 40% (12403/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2207018Z Receiving objects: 41% (12713/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2220494Z Receiving objects: 42% (13023/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2236904Z Receiving objects: 43% (13333/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2253404Z Receiving objects: 44% (13643/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2261709Z Receiving objects: 45% (13953/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2266603Z Receiving objects: 46% (14263/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2280973Z Receiving objects: 47% (14573/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2302491Z Receiving objects: 48% (14883/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2315986Z Receiving objects: 49% (15193/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2520134Z Receiving objects: 50% (15503/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2526425Z Receiving objects: 51% (15814/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2533755Z Receiving objects: 52% (16124/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2540520Z Receiving objects: 53% (16434/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2546265Z Receiving objects: 54% (16744/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2564081Z Receiving objects: 55% (17054/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2574937Z Receiving objects: 56% (17364/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2581261Z Receiving objects: 57% (17674/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2595769Z Receiving objects: 58% (17984/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2607847Z Receiving objects: 59% (18294/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2618548Z Receiving objects: 60% (18604/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2760557Z Receiving objects: 61% (18914/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2766582Z Receiving objects: 62% (19224/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2772773Z Receiving objects: 63% (19534/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2781590Z Receiving objects: 64% (19844/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2789337Z Receiving objects: 65% (20154/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.2798753Z Receiving objects: 66% (20464/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3062139Z Receiving objects: 67% (20775/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3471074Z Receiving objects: 68% (21085/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3479318Z Receiving objects: 69% (21395/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3492542Z Receiving objects: 70% (21705/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3504048Z Receiving objects: 71% (22015/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3519210Z Receiving objects: 72% (22325/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3528846Z Receiving objects: 73% (22635/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3546224Z Receiving objects: 74% (22945/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3556520Z Receiving objects: 75% (23255/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3564026Z Receiving objects: 76% (23565/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3572587Z Receiving objects: 77% (23875/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3788648Z Receiving objects: 78% (24185/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3822444Z Receiving objects: 79% (24495/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3832199Z Receiving objects: 80% (24805/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3845314Z Receiving objects: 81% (25115/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3857860Z Receiving objects: 82% (25425/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.3907305Z Receiving objects: 83% (25735/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4050120Z Receiving objects: 84% (26046/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4056275Z Receiving objects: 85% (26356/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4064377Z Receiving objects: 86% (26666/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4070074Z Receiving objects: 87% (26976/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4077906Z Receiving objects: 88% (27286/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4084310Z Receiving objects: 89% (27596/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4199324Z Receiving objects: 90% (27906/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4212828Z Receiving objects: 91% (28216/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4224626Z Receiving objects: 92% (28526/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4243866Z Receiving objects: 93% (28836/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4278635Z Receiving objects: 94% (29146/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4289248Z Receiving objects: 95% (29456/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4296332Z Receiving objects: 96% (29766/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4302989Z Receiving objects: 97% (30076/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4371464Z Receiving objects: 98% (30386/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4398508Z Receiving objects: 99% (30696/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4399561Z remote: Total 31006 (delta 1014), reused 1480 (delta 825), pack-reused 29260 (from 1) 2024-08-20T22:29:08.4416259Z Receiving objects: 100% (31006/31006), 221.71 MiB | 73.95 MiB/s 2024-08-20T22:29:08.4416947Z Receiving objects: 100% (31006/31006), 232.50 MiB | 71.95 MiB/s, done. 2024-08-20T22:29:08.4450099Z Resolving deltas: 0% (0/16734) 2024-08-20T22:29:08.4463040Z Resolving deltas: 1% (168/16734) 2024-08-20T22:29:08.4474479Z Resolving deltas: 2% (391/16734) 2024-08-20T22:29:08.4497932Z Resolving deltas: 3% (506/16734) 2024-08-20T22:29:08.4511466Z Resolving deltas: 4% (670/16734) 2024-08-20T22:29:08.4521715Z Resolving deltas: 5% (883/16734) 2024-08-20T22:29:08.4535974Z Resolving deltas: 6% (1006/16734) 2024-08-20T22:29:08.4544763Z Resolving deltas: 7% (1182/16734) 2024-08-20T22:29:08.4555213Z Resolving deltas: 8% (1363/16734) 2024-08-20T22:29:08.4565469Z Resolving deltas: 9% (1526/16734) 2024-08-20T22:29:08.4575829Z Resolving deltas: 10% (1684/16734) 2024-08-20T22:29:08.4589699Z Resolving deltas: 11% (1843/16734) 2024-08-20T22:29:08.4602501Z Resolving deltas: 12% (2011/16734) 2024-08-20T22:29:08.4615727Z Resolving deltas: 13% (2181/16734) 2024-08-20T22:29:08.4630463Z Resolving deltas: 14% (2385/16734) 2024-08-20T22:29:08.4647516Z Resolving deltas: 15% (2511/16734) 2024-08-20T22:29:08.4661503Z Resolving deltas: 16% (2688/16734) 2024-08-20T22:29:08.4697377Z Resolving deltas: 17% (2860/16734) 2024-08-20T22:29:08.4723008Z Resolving deltas: 18% (3030/16734) 2024-08-20T22:29:08.4744445Z Resolving deltas: 19% (3188/16734) 2024-08-20T22:29:08.4747261Z Resolving deltas: 20% (3481/16734) 2024-08-20T22:29:08.4762505Z Resolving deltas: 21% (3516/16734) 2024-08-20T22:29:08.4777964Z Resolving deltas: 22% (3683/16734) 2024-08-20T22:29:08.4789706Z Resolving deltas: 23% (3887/16734) 2024-08-20T22:29:08.4804876Z Resolving deltas: 24% (4037/16734) 2024-08-20T22:29:08.4842523Z Resolving deltas: 25% (4212/16734) 2024-08-20T22:29:08.4862982Z Resolving deltas: 26% (4369/16734) 2024-08-20T22:29:08.4876541Z Resolving deltas: 27% (4542/16734) 2024-08-20T22:29:08.4893407Z Resolving deltas: 28% (4688/16734) 2024-08-20T22:29:08.4910793Z Resolving deltas: 29% (4853/16734) 2024-08-20T22:29:08.4924466Z Resolving deltas: 30% (5021/16734) 2024-08-20T22:29:08.4937974Z Resolving deltas: 31% (5188/16734) 2024-08-20T22:29:08.4949858Z Resolving deltas: 32% (5393/16734) 2024-08-20T22:29:08.4961375Z Resolving deltas: 33% (5566/16734) 2024-08-20T22:29:08.4983077Z Resolving deltas: 34% (5701/16734) 2024-08-20T22:29:08.5001921Z Resolving deltas: 35% (5864/16734) 2024-08-20T22:29:08.5013680Z Resolving deltas: 36% (6045/16734) 2024-08-20T22:29:08.5035303Z Resolving deltas: 37% (6218/16734) 2024-08-20T22:29:08.5054610Z Resolving deltas: 38% (6360/16734) 2024-08-20T22:29:08.5066247Z Resolving deltas: 39% (6563/16734) 2024-08-20T22:29:08.5090946Z Resolving deltas: 40% (6702/16734) 2024-08-20T22:29:08.5098491Z Resolving deltas: 41% (6923/16734) 2024-08-20T22:29:08.5106796Z Resolving deltas: 42% (7054/16734) 2024-08-20T22:29:08.5118814Z Resolving deltas: 43% (7203/16734) 2024-08-20T22:29:08.5141459Z Resolving deltas: 44% (7374/16734) 2024-08-20T22:29:08.5156697Z Resolving deltas: 45% (7532/16734) 2024-08-20T22:29:08.5167445Z Resolving deltas: 46% (7698/16734) 2024-08-20T22:29:08.5182980Z Resolving deltas: 47% (7866/16734) 2024-08-20T22:29:08.5194231Z Resolving deltas: 48% (8036/16734) 2024-08-20T22:29:08.5199654Z Resolving deltas: 49% (8233/16734) 2024-08-20T22:29:08.5210662Z Resolving deltas: 50% (8367/16734) 2024-08-20T22:29:08.5236712Z Resolving deltas: 51% (8535/16734) 2024-08-20T22:29:08.5245159Z Resolving deltas: 53% (8874/16734) 2024-08-20T22:29:08.5258101Z Resolving deltas: 54% (9046/16734) 2024-08-20T22:29:08.5270225Z Resolving deltas: 55% (9204/16734) 2024-08-20T22:29:08.5277468Z Resolving deltas: 56% (9380/16734) 2024-08-20T22:29:08.5288143Z Resolving deltas: 57% (9549/16734) 2024-08-20T22:29:08.5298107Z Resolving deltas: 58% (9809/16734) 2024-08-20T22:29:08.5309232Z Resolving deltas: 59% (9929/16734) 2024-08-20T22:29:08.5320455Z Resolving deltas: 60% (10097/16734) 2024-08-20T22:29:08.5337651Z Resolving deltas: 61% (10226/16734) 2024-08-20T22:29:08.5343561Z Resolving deltas: 62% (10465/16734) 2024-08-20T22:29:08.5368217Z Resolving deltas: 63% (10563/16734) 2024-08-20T22:29:08.5388905Z Resolving deltas: 65% (10890/16734) 2024-08-20T22:29:08.5393652Z Resolving deltas: 66% (11174/16734) 2024-08-20T22:29:08.5411937Z Resolving deltas: 67% (11240/16734) 2024-08-20T22:29:08.5420464Z Resolving deltas: 68% (11380/16734) 2024-08-20T22:29:08.5432816Z Resolving deltas: 69% (11549/16734) 2024-08-20T22:29:08.5449280Z Resolving deltas: 70% (11714/16734) 2024-08-20T22:29:08.5461441Z Resolving deltas: 71% (11882/16734) 2024-08-20T22:29:08.5470479Z Resolving deltas: 72% (12050/16734) 2024-08-20T22:29:08.5477739Z Resolving deltas: 73% (12216/16734) 2024-08-20T22:29:08.5492209Z Resolving deltas: 74% (12385/16734) 2024-08-20T22:29:08.5497081Z Resolving deltas: 75% (12643/16734) 2024-08-20T22:29:08.5505195Z Resolving deltas: 76% (12739/16734) 2024-08-20T22:29:08.5512662Z Resolving deltas: 77% (12914/16734) 2024-08-20T22:29:08.5524532Z Resolving deltas: 78% (13054/16734) 2024-08-20T22:29:08.5561354Z Resolving deltas: 79% (13224/16734) 2024-08-20T22:29:08.5561965Z Resolving deltas: 80% (13390/16734) 2024-08-20T22:29:08.5562525Z Resolving deltas: 81% (13570/16734) 2024-08-20T22:29:08.5563030Z Resolving deltas: 82% (13722/16734) 2024-08-20T22:29:08.5569781Z Resolving deltas: 83% (13891/16734) 2024-08-20T22:29:08.5578703Z Resolving deltas: 84% (14077/16734) 2024-08-20T22:29:08.5584696Z Resolving deltas: 85% (14254/16734) 2024-08-20T22:29:08.5594715Z Resolving deltas: 86% (14397/16734) 2024-08-20T22:29:08.5602932Z Resolving deltas: 87% (14559/16734) 2024-08-20T22:29:08.5612628Z Resolving deltas: 88% (14729/16734) 2024-08-20T22:29:08.5619166Z Resolving deltas: 89% (14910/16734) 2024-08-20T22:29:08.5627587Z Resolving deltas: 90% (15075/16734) 2024-08-20T22:29:08.5636906Z Resolving deltas: 91% (15228/16734) 2024-08-20T22:29:08.5644076Z Resolving deltas: 92% (15400/16734) 2024-08-20T22:29:08.5652838Z Resolving deltas: 93% (15572/16734) 2024-08-20T22:29:08.5661450Z Resolving deltas: 94% (15736/16734) 2024-08-20T22:29:08.5674355Z Resolving deltas: 95% (15931/16734) 2024-08-20T22:29:08.5677398Z Resolving deltas: 96% (16194/16734) 2024-08-20T22:29:08.5685913Z Resolving deltas: 97% (16237/16734) 2024-08-20T22:29:08.5741590Z Resolving deltas: 98% (16403/16734) 2024-08-20T22:29:08.5802683Z Resolving deltas: 99% (16587/16734) 2024-08-20T22:29:08.5803290Z Resolving deltas: 100% (16734/16734) 2024-08-20T22:29:08.5803873Z Resolving deltas: 100% (16734/16734), done. 2024-08-20T22:29:09.5856255Z + pushd torchbench 2024-08-20T22:29:09.5857087Z ~/workspace/torchbench ~/workspace 2024-08-20T22:29:09.5857616Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:09.6644649Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T22:29:09.6645245Z 2024-08-20T22:29:09.6645792Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T22:29:09.6646623Z changes and commit them, and you can discard any commits you make in this 2024-08-20T22:29:09.6647381Z state without impacting any branches by switching back to a branch. 2024-08-20T22:29:09.6647823Z 2024-08-20T22:29:09.6648131Z If you want to create a new branch to retain commits you create, you may 2024-08-20T22:29:09.6648943Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T22:29:09.6649370Z 2024-08-20T22:29:09.6649576Z git switch -c 2024-08-20T22:29:09.6649845Z 2024-08-20T22:29:09.6649992Z Or undo this operation with: 2024-08-20T22:29:09.6650240Z 2024-08-20T22:29:09.6650374Z git switch - 2024-08-20T22:29:09.6650552Z 2024-08-20T22:29:09.6650887Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T22:29:09.6651383Z 2024-08-20T22:29:09.6651546Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T22:29:09.6656000Z + '[' '' ']' 2024-08-20T22:29:09.6656508Z + python install.py --continue_on_fail 2024-08-20T22:29:11.9577755Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T22:29:30.4217173Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T22:29:33.5342674Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T22:29:35.4284334Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T22:29:37.3406556Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T22:29:37.3643628Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T22:29:44.9443149Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T22:29:47.6654841Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T22:29:50.3811992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T22:29:53.1275885Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T22:29:53.1279147Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T22:29:54.1748301Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T22:29:56.9303911Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T22:29:56.9538609Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T22:30:54.3254925Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T22:31:03.0188398Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T22:31:10.2424299Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T22:31:17.1055674Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T22:31:25.6199552Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T22:31:32.6912739Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T22:31:39.8281801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T22:31:46.7988609Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T22:31:53.8244394Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T22:32:01.3817499Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T22:32:08.2567847Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T22:32:15.3202844Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T22:32:16.5210146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T22:32:52.8540203Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T22:32:57.4607238Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T22:33:02.6911176Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:33:14.0749108Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:33:15.2615549Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:33:16.9362861Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:33:23.6128641Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:33:30.7112651Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:33:36.1943713Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:33:43.9504179Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:33:49.8417968Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:33:54.8486221Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:34:01.0452672Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:34:17.8337709Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:34:24.0608186Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:34:27.9903166Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:34:32.8975529Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:34:39.6928270Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:34:39.6932488Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:34:52.7534649Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:34:56.6910037Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:34:56.6912333Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:35:04.3706836Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:35:05.6001768Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:35:06.8390730Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:35:51.5742760Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:37:20.9531396Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:37:20.9821657Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:37:22.5344463Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:37:22.5590604Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:37:22.5836041Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:37:22.6076051Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:37:22.6316139Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:37:22.6557367Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:37:43.8562643Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:37:43.8566596Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:37:45.1129502Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:37:47.5620235Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:37:47.5863513Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:37:47.6102597Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:37:47.6348150Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:37:47.6598647Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:37:47.6841304Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:37:54.3303968Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:37:55.7599051Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:37:59.2630225Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:37:59.2871961Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:37:59.3119316Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:37:59.3359146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:37:59.3601297Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:37:59.3839330Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:38:14.4503809Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:38:30.8414608Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:38:30.8658636Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:38:30.8662318Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:38:30.8663897Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:38:33.2796057Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:38:35.0432195Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:38:35.0677966Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:39:00.9932358Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:39:07.5644593Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:39:12.0018295Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:39:18.5799127Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:39:18.6051047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:39:18.6300739Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:39:18.6542648Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:39:18.6783438Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:39:18.7033087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:39:18.7274983Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:39:18.7520812Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:39:28.5494242Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:39:39.8772721Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:39:39.9028014Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:39:42.0054771Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:39:43.9867082Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:39:44.4849436Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:39:44.4850092Z Print all dependencies after TorchBench is installed 2024-08-20T22:39:44.4850630Z + python -mpip freeze 2024-08-20T22:39:44.9908294Z absl-py==2.1.0 2024-08-20T22:39:44.9908710Z accelerate==0.33.0 2024-08-20T22:39:44.9912235Z aiohappyeyeballs==2.4.0 2024-08-20T22:39:44.9912747Z aiohttp==3.10.5 2024-08-20T22:39:44.9913173Z aiosignal==1.3.1 2024-08-20T22:39:44.9913735Z annotated-types==0.7.0 2024-08-20T22:39:44.9917041Z antlr4-python3-runtime==4.9.3 2024-08-20T22:39:44.9918569Z anyascii==0.3.2 2024-08-20T22:39:44.9918978Z astroid==3.2.4 2024-08-20T22:39:44.9919359Z astunparse==1.6.3 2024-08-20T22:39:44.9919880Z async-timeout==4.0.3 2024-08-20T22:39:44.9920325Z attrs==23.1.0 2024-08-20T22:39:44.9920695Z audioread==3.0.1 2024-08-20T22:39:44.9921059Z babel==2.16.0 2024-08-20T22:39:44.9921389Z beautifulsoup4==4.12.3 2024-08-20T22:39:44.9922581Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:39:44.9923643Z bibtexparser==2.0.0b7 2024-08-20T22:39:44.9923984Z black==24.8.0 2024-08-20T22:39:44.9924324Z blinker==1.8.2 2024-08-20T22:39:44.9924691Z blis==0.7.11 2024-08-20T22:39:44.9925062Z blobfile==2.1.1 2024-08-20T22:39:44.9925469Z bokeh==3.5.1 2024-08-20T22:39:44.9925835Z boto3==1.19.12 2024-08-20T22:39:44.9926206Z botocore==1.22.12 2024-08-20T22:39:44.9926571Z bs4==0.0.2 2024-08-20T22:39:44.9926927Z cachetools==5.5.0 2024-08-20T22:39:44.9927324Z cardboardlint==1.3.1 2024-08-20T22:39:44.9929555Z catalogue==2.0.10 2024-08-20T22:39:44.9930055Z certifi==2024.7.4 2024-08-20T22:39:44.9930500Z cffi==1.17.0 2024-08-20T22:39:44.9932559Z charset-normalizer==3.3.2 2024-08-20T22:39:44.9932954Z click==8.1.7 2024-08-20T22:39:44.9933253Z clldutils==3.22.2 2024-08-20T22:39:44.9933906Z cloudpathlib==0.18.1 2024-08-20T22:39:44.9934353Z cloudpickle==3.0.0 2024-08-20T22:39:44.9934766Z colorama==0.4.6 2024-08-20T22:39:44.9935163Z colorlog==6.8.2 2024-08-20T22:39:44.9935503Z confection==0.1.5 2024-08-20T22:39:44.9935818Z contourpy==1.2.1 2024-08-20T22:39:44.9936154Z coremltools==5.0b5 2024-08-20T22:39:44.9936476Z cryptography==43.0.0 2024-08-20T22:39:44.9936790Z csvw==3.3.0 2024-08-20T22:39:44.9937077Z cycler==0.12.1 2024-08-20T22:39:44.9937369Z cymem==2.0.8 2024-08-20T22:39:44.9937718Z DALL-E==0.1 2024-08-20T22:39:44.9938001Z datasets==2.21.0 2024-08-20T22:39:44.9938311Z decorator==5.1.1 2024-08-20T22:39:44.9938624Z defusedxml==0.7.1 2024-08-20T22:39:44.9938931Z Deprecated==1.2.14 2024-08-20T22:39:44.9939705Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:39:44.9940719Z diffusers==0.20.2 2024-08-20T22:39:44.9941146Z dill==0.3.8 2024-08-20T22:39:44.9941519Z diskcache==5.6.3 2024-08-20T22:39:44.9941908Z distro==1.9.0 2024-08-20T22:39:44.9942280Z dlinfo==1.2.1 2024-08-20T22:39:44.9942722Z docker-pycreds==0.4.0 2024-08-20T22:39:44.9943145Z dominate==2.9.1 2024-08-20T22:39:44.9944183Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:39:44.9944977Z einops==0.8.0 2024-08-20T22:39:44.9945273Z exceptiongroup==1.2.2 2024-08-20T22:39:44.9945602Z execnet==2.1.1 2024-08-20T22:39:44.9945897Z expecttest==0.1.6 2024-08-20T22:39:44.9946201Z FastNLP==0.6.0 2024-08-20T22:39:44.9946546Z ffmpeg-python==0.2.0 2024-08-20T22:39:44.9946873Z filelock==3.13.1 2024-08-20T22:39:44.9947315Z Flask==3.0.3 2024-08-20T22:39:44.9947598Z flatbuffers==2.0 2024-08-20T22:39:44.9947996Z fonttools==4.53.1 2024-08-20T22:39:44.9948308Z frozenlist==1.4.1 2024-08-20T22:39:44.9948610Z fsspec==2024.6.1 2024-08-20T22:39:44.9948899Z ftfy==6.2.3 2024-08-20T22:39:44.9949521Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:39:44.9950233Z future==1.0.0 2024-08-20T22:39:44.9950535Z fvcore==0.1.5.post20221221 2024-08-20T22:39:44.9950884Z gdown==5.2.0 2024-08-20T22:39:44.9951162Z geojson==2.5.0 2024-08-20T22:39:44.9951452Z ghstack==0.8.0 2024-08-20T22:39:44.9951740Z gitdb==4.0.11 2024-08-20T22:39:44.9952028Z GitPython==3.1.43 2024-08-20T22:39:44.9952378Z google-auth==2.34.0 2024-08-20T22:39:44.9952746Z google-auth-oauthlib==1.0.0 2024-08-20T22:39:44.9953105Z grpcio==1.65.5 2024-08-20T22:39:44.9953383Z gym==0.26.2 2024-08-20T22:39:44.9953695Z gym-notices==0.0.8 2024-08-20T22:39:44.9953995Z h5py==3.11.0 2024-08-20T22:39:44.9954276Z higher==0.2.1 2024-08-20T22:39:44.9954605Z huggingface-hub==0.24.6 2024-08-20T22:39:44.9954970Z hydra-core==1.3.2 2024-08-20T22:39:44.9955551Z hypothesis==5.35.1 2024-08-20T22:39:44.9955847Z idna==3.7 2024-08-20T22:39:44.9956121Z imageio==2.35.1 2024-08-20T22:39:44.9956433Z importlib_metadata==8.4.0 2024-08-20T22:39:44.9956772Z inflect==7.3.1 2024-08-20T22:39:44.9957056Z iniconfig==2.0.0 2024-08-20T22:39:44.9957352Z iopath==0.1.9 2024-08-20T22:39:44.9957641Z isodate==0.6.1 2024-08-20T22:39:44.9957921Z isort==5.13.2 2024-08-20T22:39:44.9958210Z itsdangerous==2.2.0 2024-08-20T22:39:44.9958514Z Jinja2==3.1.4 2024-08-20T22:39:44.9958800Z jmespath==0.10.0 2024-08-20T22:39:44.9959087Z joblib==1.4.2 2024-08-20T22:39:44.9959377Z jsonpatch==1.33 2024-08-20T22:39:44.9959675Z jsonpointer==3.0.0 2024-08-20T22:39:44.9959982Z jsonschema==4.23.0 2024-08-20T22:39:44.9960374Z jsonschema-specifications==2023.12.1 2024-08-20T22:39:44.9960788Z junitparser==2.1.1 2024-08-20T22:39:44.9961126Z kaldi-io==0.9.8 2024-08-20T22:39:44.9961419Z kiwisolver==1.4.5 2024-08-20T22:39:44.9961715Z kornia==0.7.3 2024-08-20T22:39:44.9961999Z kornia_rs==0.1.5 2024-08-20T22:39:44.9962308Z lameenc==1.7.0 2024-08-20T22:39:44.9962593Z langcodes==3.4.0 2024-08-20T22:39:44.9962890Z langdetect==1.0.9 2024-08-20T22:39:44.9963388Z language-tags==1.2.0 2024-08-20T22:39:44.9963709Z language_data==1.2.0 2024-08-20T22:39:44.9964030Z lark==0.12.0 2024-08-20T22:39:44.9964314Z lazy_loader==0.4 2024-08-20T22:39:44.9964606Z libcst==1.4.0 2024-08-20T22:39:44.9964885Z librosa==0.9.2 2024-08-20T22:39:44.9965173Z lintrunner==0.12.5 2024-08-20T22:39:44.9965474Z llvmlite==0.38.1 2024-08-20T22:39:44.9965760Z lxml==4.9.4 2024-08-20T22:39:44.9966084Z marisa-trie==1.2.0 2024-08-20T22:39:44.9966388Z Markdown==3.7 2024-08-20T22:39:44.9966718Z markdown-it-py==3.0.0 2024-08-20T22:39:44.9967039Z MarkupSafe==2.1.5 2024-08-20T22:39:44.9967341Z matplotlib==3.8.4 2024-08-20T22:39:44.9967641Z mccabe==0.7.0 2024-08-20T22:39:44.9967916Z mdurl==0.1.2 2024-08-20T22:39:44.9968233Z ml-dtypes==0.4.0 2024-08-20T22:39:44.9968534Z MonkeyType==23.3.0 2024-08-20T22:39:44.9968880Z more-itertools==10.4.0 2024-08-20T22:39:44.9969204Z mpmath==1.3.0 2024-08-20T22:39:44.9969495Z msgpack==1.0.8 2024-08-20T22:39:44.9969787Z multidict==6.0.5 2024-08-20T22:39:44.9970087Z multiprocess==0.70.16 2024-08-20T22:39:44.9970416Z murmurhash==1.0.10 2024-08-20T22:39:44.9970722Z musdb==0.4.2 2024-08-20T22:39:44.9971002Z museval==0.4.1 2024-08-20T22:39:44.9971282Z mypy==1.10.0 2024-08-20T22:39:44.9971617Z mypy-extensions==1.0.0 2024-08-20T22:39:44.9971944Z networkx==2.8.8 2024-08-20T22:39:44.9972226Z ninja==1.10.2 2024-08-20T22:39:44.9972512Z nose==1.3.7 2024-08-20T22:39:44.9972782Z numba==0.55.2 2024-08-20T22:39:44.9973062Z numpy==1.21.2 2024-08-20T22:39:44.9973407Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:39:44.9973841Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:39:44.9974288Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:39:44.9974740Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:39:44.9975175Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:39:44.9975582Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:39:44.9975998Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:39:44.9976425Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:39:44.9976914Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:39:44.9977347Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:39:44.9977748Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:39:44.9978159Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:39:44.9978505Z oauthlib==3.2.2 2024-08-20T22:39:44.9978799Z omegaconf==2.3.0 2024-08-20T22:39:44.9979089Z onnx==1.16.1 2024-08-20T22:39:44.9979400Z onnxscript==0.1.0.dev20240817 2024-08-20T22:39:44.9979757Z opacus==1.5.2 2024-08-20T22:39:44.9980091Z opencv-python==4.8.0.74 2024-08-20T22:39:44.9980452Z opt-einsum==3.3.0 2024-08-20T22:39:44.9980752Z optree==0.12.1 2024-08-20T22:39:44.9981045Z packaging==24.1 2024-08-20T22:39:44.9981341Z pandas==2.0.3 2024-08-20T22:39:44.9981623Z patch==1.16 2024-08-20T22:39:44.9981899Z pathspec==0.12.1 2024-08-20T22:39:44.9982203Z phonemizer==3.3.0 2024-08-20T22:39:44.9982497Z pillow==10.3.0 2024-08-20T22:39:44.9982790Z platformdirs==4.2.2 2024-08-20T22:39:44.9983227Z pluggy==1.5.0 2024-08-20T22:39:44.9983512Z pooch==1.8.2 2024-08-20T22:39:44.9983798Z portalocker==2.10.1 2024-08-20T22:39:44.9984107Z preshed==3.0.9 2024-08-20T22:39:44.9984409Z prettytable==3.11.0 2024-08-20T22:39:44.9984712Z protobuf==3.20.2 2024-08-20T22:39:44.9985010Z psutil==6.0.0 2024-08-20T22:39:44.9985337Z py-cpuinfo==9.0.0 2024-08-20T22:39:44.9985639Z pyaml==24.7.0 2024-08-20T22:39:44.9985917Z pyarrow==17.0.0 2024-08-20T22:39:44.9986205Z pyasn1==0.6.0 2024-08-20T22:39:44.9986496Z pyasn1_modules==0.4.0 2024-08-20T22:39:44.9986822Z pyclipper==1.3.0.post5 2024-08-20T22:39:44.9987305Z pycocotools==2.0.8 2024-08-20T22:39:44.9987623Z pycparser==2.22 2024-08-20T22:39:44.9987932Z pycryptodomex==3.20.0 2024-08-20T22:39:44.9988257Z pydantic==2.8.2 2024-08-20T22:39:44.9988560Z pydantic_core==2.20.1 2024-08-20T22:39:44.9988879Z pydot==3.0.1 2024-08-20T22:39:44.9989160Z pygame==2.6.0 2024-08-20T22:39:44.9989460Z PyGithub==2.3.0 2024-08-20T22:39:44.9989762Z Pygments==2.15.0 2024-08-20T22:39:44.9990066Z PyJWT==2.9.0 2024-08-20T22:39:44.9990364Z pylatexenc==2.10 2024-08-20T22:39:44.9990665Z pylint==3.2.6 2024-08-20T22:39:44.9990953Z PyNaCl==1.5.0 2024-08-20T22:39:44.9991341Z pynvml==11.5.3 2024-08-20T22:39:44.9991652Z pyparsing==3.1.2 2024-08-20T22:39:44.9991959Z pypdfium2==4.30.0 2024-08-20T22:39:44.9992264Z pysbd==0.3.4 2024-08-20T22:39:44.9992546Z PySocks==1.7.1 2024-08-20T22:39:44.9992839Z pytest==7.3.2 2024-08-20T22:39:44.9993195Z pytest-benchmark==4.0.0 2024-08-20T22:39:44.9993569Z pytest-cpp==2.3.0 2024-08-20T22:39:44.9993930Z pytest-flakefinder==1.1.0 2024-08-20T22:39:44.9994344Z pytest-rerunfailures==14.0 2024-08-20T22:39:44.9994738Z pytest-xdist==3.3.1 2024-08-20T22:39:44.9995106Z python-dateutil==2.9.0.post0 2024-08-20T22:39:44.9995905Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:39:44.9997396Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:39:44.9998384Z pytz==2024.1 2024-08-20T22:39:44.9998670Z PyWavelets==1.4.1 2024-08-20T22:39:44.9999050Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:39:44.9999512Z rapidfuzz==3.9.6 2024-08-20T22:39:44.9999814Z rdflib==7.0.0 2024-08-20T22:39:45.0000096Z redis==5.0.8 2024-08-20T22:39:45.0000385Z referencing==0.35.1 2024-08-20T22:39:45.0000698Z regex==2024.7.24 2024-08-20T22:39:45.0000990Z requests==2.32.3 2024-08-20T22:39:45.0001349Z requests-oauthlib==2.0.0 2024-08-20T22:39:45.0001696Z resampy==0.4.3 2024-08-20T22:39:45.0001989Z rfc3986==1.5.0 2024-08-20T22:39:45.0002269Z rich==13.7.1 2024-08-20T22:39:45.0002552Z rockset==1.0.3 2024-08-20T22:39:45.0002879Z rpds-py==0.20.0 2024-08-20T22:39:45.0003165Z rsa==4.9 2024-08-20T22:39:45.0003444Z s3transfer==0.5.2 2024-08-20T22:39:45.0003753Z safetensors==0.4.4 2024-08-20T22:39:45.0004103Z scikit-image==0.21.0 2024-08-20T22:39:45.0004457Z scikit-learn==1.5.1 2024-08-20T22:39:45.0004768Z scipy==1.10.1 2024-08-20T22:39:45.0005058Z segments==2.2.1 2024-08-20T22:39:45.0005361Z sentencepiece==0.2.0 2024-08-20T22:39:45.0005729Z sentry-sdk==2.13.0 2024-08-20T22:39:45.0006039Z setproctitle==1.3.3 2024-08-20T22:39:45.0006369Z shapely==2.0.6 2024-08-20T22:39:45.0006709Z shellingham==1.5.4 2024-08-20T22:39:45.0007025Z simplejson==3.19.3 2024-08-20T22:39:45.0007423Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:39:45.0007927Z smart-open==7.0.4 2024-08-20T22:39:45.0008230Z smmap==5.0.1 2024-08-20T22:39:45.0008533Z sortedcontainers==2.4.0 2024-08-20T22:39:45.0008869Z soundfile==0.12.1 2024-08-20T22:39:45.0009169Z soupsieve==2.6 2024-08-20T22:39:45.0009460Z soxr==0.4.0 2024-08-20T22:39:45.0009736Z spacy==3.7.6 2024-08-20T22:39:45.0010062Z spacy-legacy==3.0.12 2024-08-20T22:39:45.0010429Z spacy-loggers==1.0.5 2024-08-20T22:39:45.0010747Z srsly==2.4.8 2024-08-20T22:39:45.0011034Z stempeg==0.2.3 2024-08-20T22:39:45.0011325Z submitit==1.5.1 2024-08-20T22:39:45.0011620Z sympy==1.13.1 2024-08-20T22:39:45.0011905Z tabulate==0.9.0 2024-08-20T22:39:45.0012409Z tb-nightly==2.13.0a20230426 2024-08-20T22:39:45.0012764Z tensorboard==2.13.0 2024-08-20T22:39:45.0013149Z tensorboard-data-server==0.7.2 2024-08-20T22:39:45.0013540Z tensorboardX==2.6.2.2 2024-08-20T22:39:45.0013861Z termcolor==2.4.0 2024-08-20T22:39:45.0014163Z thinc==8.2.5 2024-08-20T22:39:45.0014460Z threadpoolctl==3.5.0 2024-08-20T22:39:45.0014780Z tifffile==2024.8.10 2024-08-20T22:39:45.0015595Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:39:45.0016388Z tlparse==0.3.25 2024-08-20T22:39:45.0016686Z tokenizers==0.15.2 2024-08-20T22:39:45.0016982Z tomli==2.0.1 2024-08-20T22:39:45.0017263Z tomlkit==0.13.2 2024-08-20T22:39:45.0018470Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b444e33ace8c516878aca2862d707788a4c01d8c6190fb13e010fb9a131a57c5 2024-08-20T22:39:45.0020112Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:39:45.0021196Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:39:45.0022325Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:39:45.0023350Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:39:45.0024479Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:39:45.0025591Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:39:45.0026639Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:39:45.0027478Z tornado==6.4.1 2024-08-20T22:39:45.0027774Z tqdm==4.66.5 2024-08-20T22:39:45.0028069Z transformers==4.38.1 2024-08-20T22:39:45.0028387Z treetable==0.2.5 2024-08-20T22:39:45.0029215Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:39:45.0030025Z typeguard==4.3.0 2024-08-20T22:39:45.0030325Z typer==0.12.4 2024-08-20T22:39:45.0030630Z typing_extensions==4.12.2 2024-08-20T22:39:45.0030977Z tzdata==2024.1 2024-08-20T22:39:45.0031274Z Unidecode==1.3.8 2024-08-20T22:39:45.0031651Z unittest-xml-reporting==3.2.0 2024-08-20T22:39:45.0032028Z uritemplate==4.1.1 2024-08-20T22:39:45.0032343Z urllib3==1.26.19 2024-08-20T22:39:45.0032646Z visdom==0.2.4 2024-08-20T22:39:45.0032927Z wandb==0.17.7 2024-08-20T22:39:45.0033212Z wasabi==1.1.3 2024-08-20T22:39:45.0033954Z wcwidth==0.2.13 2024-08-20T22:39:45.0034249Z weasel==0.4.1 2024-08-20T22:39:45.0034605Z websocket-client==1.8.0 2024-08-20T22:39:45.0034935Z Werkzeug==3.0.3 2024-08-20T22:39:45.0035230Z wrapt==1.16.0 2024-08-20T22:39:45.0035507Z xdoctest==1.1.0 2024-08-20T22:39:45.0035796Z xxhash==3.5.0 2024-08-20T22:39:45.0036086Z xyzservices==2024.6.0 2024-08-20T22:39:45.0036400Z yacs==0.1.8 2024-08-20T22:39:45.0036726Z yarl==1.9.4 2024-08-20T22:39:45.0037037Z z3-solver==4.12.2.0 2024-08-20T22:39:45.0037341Z zipp==3.20.0 2024-08-20T22:39:45.0232026Z + popd 2024-08-20T22:39:45.0232373Z ~/workspace 2024-08-20T22:39:45.0232696Z + [[ inductor_torchbench != *cpu* ]] 2024-08-20T22:39:45.0233213Z + install_torchrec_and_fbgemm 2024-08-20T22:39:45.0233892Z + local torchrec_commit 2024-08-20T22:39:45.0235661Z ++ get_pinned_commit torchrec 2024-08-20T22:39:45.0236083Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-08-20T22:39:45.0330562Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T22:39:45.0331101Z + local fbgemm_commit 2024-08-20T22:39:45.0333661Z ++ get_pinned_commit fbgemm 2024-08-20T22:39:45.0334139Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-08-20T22:39:45.0405901Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:45.0406713Z + pip_uninstall torchrec-nightly 2024-08-20T22:39:45.0407293Z + pip uninstall -y torchrec-nightly 2024-08-20T22:39:45.4093151Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-08-20T22:39:45.4451542Z + pip_uninstall fbgemm-gpu-nightly 2024-08-20T22:39:45.4452202Z + pip uninstall -y fbgemm-gpu-nightly 2024-08-20T22:39:45.8170316Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-08-20T22:39:45.8530868Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:45.8531846Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:46.3548347Z Collecting setuptools-git-versioning 2024-08-20T22:39:46.3714544Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-08-20T22:39:46.4801716Z Collecting scikit-build 2024-08-20T22:39:46.4830467Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-08-20T22:39:46.5992407Z Collecting pyre-extensions 2024-08-20T22:39:46.6022756Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-08-20T22:39:46.6515467Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-08-20T22:39:46.6518462Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0) 2024-08-20T22:39:46.6983090Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-08-20T22:39:46.7013896Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-08-20T22:39:46.7495880Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-08-20T22:39:46.7500293Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-08-20T22:39:46.7503923Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-08-20T22:39:46.8056656Z Collecting typing-inspect (from pyre-extensions) 2024-08-20T22:39:46.8084386Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-08-20T22:39:46.8590936Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-08-20T22:39:46.8678874Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-08-20T22:39:46.8717567Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-08-20T22:39:46.9250516Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-08-20T22:39:46.9824921Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-08-20T22:39:47.0348226Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-08-20T22:39:47.0813452Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-08-20T22:39:47.9685832Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-08-20T22:39:48.3230114Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-08-20T22:39:48.4364136Z + CUDA_PATH=/usr/local/cuda-12.1 2024-08-20T22:39:48.4365328Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:48.4367206Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:48.8089315Z Collecting fbgemm-gpu 2024-08-20T22:39:48.8091819Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-1hiiuk3_/fbgemm-gpu_9f854c6bee5b41a39763b51acddb7b07 2024-08-20T22:39:48.8122662Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-1hiiuk3_/fbgemm-gpu_9f854c6bee5b41a39763b51acddb7b07 2024-08-20T22:39:49.8802326Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-08-20T22:39:49.8829304Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:50.3748823Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:50.8879907Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:50.8881198Z Running command git submodule update --init --recursive -q 2024-08-20T22:40:00.4717636Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:40:00.4737418Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-08-20T22:40:00.4744864Z Building wheels for collected packages: fbgemm-gpu 2024-08-20T23:25:18.6799097Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T23:25:18.8630522Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252620511 sha256=0a32ad200b31d120f7bfb3967291fa5467799d3ec1e70d7f6c80af94c3c55606 2024-08-20T23:25:18.8632192Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-08-20T23:25:18.8652992Z Successfully built fbgemm-gpu 2024-08-20T23:25:19.7070601Z Installing collected packages: fbgemm-gpu 2024-08-20T23:25:23.0640262Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-08-20T23:25:23.9797646Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:23.9799205Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:24.3504992Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:24.3508920Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-zu_dhvuh 2024-08-20T23:25:24.3538352Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-zu_dhvuh 2024-08-20T23:25:25.1223335Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-08-20T23:25:25.1252621Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:25.4299021Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:25.7730597Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:26.0469458Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T23:25:26.0508853Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-08-20T23:25:26.0513889Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-08-20T23:25:26.0516341Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-08-20T23:25:26.1004814Z Collecting torchmetrics (from torchrec==0.3.2) 2024-08-20T23:25:26.1187267Z Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB) 2024-08-20T23:25:26.1583703Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5) 2024-08-20T23:25:26.1587037Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-08-20T23:25:26.1598403Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-08-20T23:25:26.1609204Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1) 2024-08-20T23:25:26.1875563Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-08-20T23:25:26.1878527Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:25:26.1883004Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:25:26.1906154Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-08-20T23:25:26.1910800Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6) 2024-08-20T23:25:26.2063994Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-08-20T23:25:26.2100023Z Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) 2024-08-20T23:25:26.2197194Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0) 2024-08-20T23:25:26.2200716Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-08-20T23:25:26.2222797Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-08-20T23:25:26.2256899Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-08-20T23:25:26.2261165Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-08-20T23:25:26.2265098Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-08-20T23:25:26.2268518Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1) 2024-08-20T23:25:26.2273760Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1) 2024-08-20T23:25:26.2289103Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-08-20T23:25:26.2706192Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-08-20T23:25:26.4088772Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB) 2024-08-20T23:25:26.4300721Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB) 2024-08-20T23:25:26.4400223Z Building wheels for collected packages: torchrec 2024-08-20T23:25:26.9567598Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-08-20T23:25:26.9575562Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=54aa4693829b64164ebed65025aa7f5275e373b4f41f0dc94cee8e6306459088 2024-08-20T23:25:26.9577603Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-08-20T23:25:26.9599756Z Successfully built torchrec 2024-08-20T23:25:27.7410573Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-08-20T23:25:28.3514973Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2 2024-08-20T23:25:28.4936884Z ++ pwd 2024-08-20T23:25:28.4938091Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T23:25:28.4938610Z + test_dynamo_benchmark torchbench 1 2024-08-20T23:25:28.4942706Z ++ pwd 2024-08-20T23:25:28.4946313Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:28.4947035Z + local suite=torchbench 2024-08-20T23:25:28.4949603Z + shift 2024-08-20T23:25:28.4950041Z + local shard_id=1 2024-08-20T23:25:28.4950511Z + shift 2024-08-20T23:25:28.4950950Z + [[ inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:28.4951513Z + [[ inductor_torchbench == *perf* ]] 2024-08-20T23:25:28.4952011Z + [[ inductor_torchbench == *cpu* ]] 2024-08-20T23:25:28.4954466Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:28.4955454Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-08-20T23:25:28.4956028Z ++ pwd 2024-08-20T23:25:28.4956487Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:28.4957203Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:28.4996034Z + local name=inference 2024-08-20T23:25:28.4996536Z + shift 2024-08-20T23:25:28.4996905Z + local suite=torchbench 2024-08-20T23:25:28.4997311Z + shift 2024-08-20T23:25:28.4997583Z + local shard_id=1 2024-08-20T23:25:28.4997886Z + shift 2024-08-20T23:25:28.4998622Z + partition_flags=() 2024-08-20T23:25:28.4999086Z + local partition_flags 2024-08-20T23:25:28.4999621Z + [[ -n 2 ]] 2024-08-20T23:25:28.5000052Z + [[ -n 1 ]] 2024-08-20T23:25:28.5000848Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:25:28.5001757Z + [[ inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:28.5002316Z + [[ inductor_torchbench == *perf* ]] 2024-08-20T23:25:28.5003443Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:28.5004061Z + [[ inductor_torchbench == *_avx2* ]] 2024-08-20T23:25:28.5004590Z + [[ inductor_torchbench == *_avx512* ]] 2024-08-20T23:25:28.5006222Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T23:25:34.4383729Z 2024-08-20T23:25:35.2876764Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:35.2877199Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:35.2910194Z cuda eval llama 2024-08-20T23:25:56.2603170Z pass 2024-08-20T23:25:56.2848457Z TIMING: code_gen:5.0587 inductor_compile:9.93224 backend_compile:16.09579 entire_frame_compile:18.99447 total_wall_time:18.99447 2024-08-20T23:25:56.2850036Z STATS: call_* op count: 1058 | FakeTensorMode.__torch_dispatch__:27613 | FakeTensor.__torch_dispatch__:2607 | ProxyTorchDispatchMode.__torch_dispatch__:10235 2024-08-20T23:25:56.2851164Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:00.7156146Z 2024-08-20T23:26:01.0183491Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:01.0183976Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:01.0186924Z cuda eval llama_v2_7b_16h 2024-08-20T23:26:01.0208556Z Traceback (most recent call last): 2024-08-20T23:26:01.0209472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:26:01.0210300Z ) = runner.load_model( 2024-08-20T23:26:01.0211244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:26:01.0212218Z benchmark = benchmark_cls( 2024-08-20T23:26:01.0213496Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:26:01.0214338Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:26:01.0215252Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-08-20T23:26:01.0216764Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:01.0218110Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:26:01.0219307Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:01.0220295Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:26:01.0221066Z self._skip_by_device_name() 2024-08-20T23:26:01.0221886Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:26:01.0223148Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:26:01.0224310Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-08-20T23:26:01.0224938Z 2024-08-20T23:26:01.0225058Z model_fail_to_load 2024-08-20T23:26:04.4843119Z 2024-08-20T23:26:04.7881940Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:04.7882961Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:04.7883982Z cuda eval llava 2024-08-20T23:26:04.7889783Z Traceback (most recent call last): 2024-08-20T23:26:04.7890851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:26:04.7891785Z ) = runner.load_model( 2024-08-20T23:26:04.7892526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:26:04.7893344Z benchmark = benchmark_cls( 2024-08-20T23:26:04.7894369Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:26:04.7895218Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:26:04.7896076Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:26:04.7897195Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:04.7898438Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:26:04.7899632Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:26:04.7900624Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:26:04.7901670Z self._skip_by_device_name() 2024-08-20T23:26:04.7902483Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:26:04.7903703Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:26:04.7904789Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:26:04.7905362Z 2024-08-20T23:26:04.7905481Z model_fail_to_load 2024-08-20T23:26:08.2412794Z 2024-08-20T23:26:08.9709476Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:08.9709955Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:08.9715265Z cuda eval maml 2024-08-20T23:26:08.9718447Z pass_due_to_skip 2024-08-20T23:26:08.9726951Z TIMING: 2024-08-20T23:26:08.9727350Z STATS: call_* op count: 0 2024-08-20T23:26:08.9727936Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:12.5365113Z 2024-08-20T23:26:13.0891955Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:13.0892379Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:13.0897554Z cuda eval maml_omniglot 2024-08-20T23:26:17.3144457Z pass 2024-08-20T23:26:17.3146344Z TIMING: code_gen:0.973 inductor_compile:1.53486 backend_compile:2.67367 entire_frame_compile:3.07579 total_wall_time:3.07579 2024-08-20T23:26:17.3147836Z STATS: call_* op count: 28 | FakeTensorMode.__torch_dispatch__:1958 | ProxyTorchDispatchMode.__torch_dispatch__:543 | FakeTensor.__torch_dispatch__:176 2024-08-20T23:26:17.3148975Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:21.1411817Z 2024-08-20T23:26:21.3592058Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-08-20T23:26:21.3758477Z 2024-08-20T23:26:21.3758537Z 2024-08-20T23:26:21.4182247Z 0% 0.00/16.9M [00:00 will be ignored 2024-08-20T23:27:53.1481409Z [rank0]:W0820 23:27:53.147000 33712 torch/_dynamo/backends/distributed.py:89] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:27:53.1714759Z [rank0]:W0820 23:27:53.170000 33712 torch/_dynamo/backends/distributed.py:106] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:27:53.1718636Z [rank0]:W0820 23:27:53.170000 33712 torch/_dynamo/backends/distributed.py:106] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:27:53.1720248Z [rank0]:W0820 23:27:53.170000 33712 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:27:53.1721728Z [rank0]:W0820 23:27:53.170000 33712 torch/_dynamo/backends/distributed.py:106] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:27:53.1723207Z [rank0]:W0820 23:27:53.170000 33712 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ 0 │ 470 │ 44910720 │ 2024-08-20T23:27:53.1724665Z [rank0]:W0820 23:27:53.170000 33712 torch/_dynamo/backends/distributed.py:106] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:28:14.3997631Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 2024-08-20T23:28:14.3999195Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 130, in forward 2024-08-20T23:28:14.4000621Z self._momentum_update_key_encoder() # update the key encoder 2024-08-20T23:28:14.4002437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:28:14.4003659Z return func(*args, **kwargs) 2024-08-20T23:28:14.4004966Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 50, in _momentum_update_key_encoder 2024-08-20T23:28:14.4008896Z param_k.mul_(self.m).add_(param_q.mul(1. - self.m)) 2024-08-20T23:28:14.4009263Z 2024-08-20T23:28:14.5145253Z [rank0]:W0820 23:28:14.513000 33712 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-20T23:28:14.5250804Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-08-20T23:28:14.5252079Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-08-20T23:28:14.5253075Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-08-20T23:28:14.5253439Z 2024-08-20T23:28:17.3981904Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:28:17.3983478Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:28:17.3984567Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] or: 2024-08-20T23:28:17.3986075Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:28:17.3987461Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] to include these operations in the captured graph. 2024-08-20T23:28:17.3988640Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:28:17.3989596Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break: from user code at: 2024-08-20T23:28:17.3991332Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:28:17.3993050Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] self._dequeue_and_enqueue(k) 2024-08-20T23:28:17.3994974Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:28:17.3996520Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] return func(*args, **kwargs) 2024-08-20T23:28:17.3998152Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:28:17.3999767Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] ptr = int(self.queue_ptr) 2024-08-20T23:28:17.4000707Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:28:17.4001496Z [rank0]:W0820 23:28:17.397000 33712 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:28:33.4951849Z pass 2024-08-20T23:28:33.4963479Z TIMING: entire_frame_compile:48.64187 code_gen:9.50576 inductor_compile:17.31986 backend_compile:28.88508 total_wall_time:48.64187 2024-08-20T23:28:33.4964902Z STATS: call_* op count: 879 | FakeTensorMode.__torch_dispatch__:39553 | FakeTensor.__torch_dispatch__:7974 | ProxyTorchDispatchMode.__torch_dispatch__:11976 2024-08-20T23:28:33.4966032Z Dynamo produced 7 graphs covering 879 ops with 5 graph breaks (3 unique) 2024-08-20T23:28:34.5441899Z [rank0]:[W820 23:28:34.000160367 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:28:39.3093970Z 2024-08-20T23:28:39.6135324Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:28:39.6135931Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:28:39.6136438Z cuda eval moondream 2024-08-20T23:28:39.6143488Z Traceback (most recent call last): 2024-08-20T23:28:39.6144395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:28:39.6145083Z ) = runner.load_model( 2024-08-20T23:28:39.6145744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:28:39.6146535Z benchmark = benchmark_cls( 2024-08-20T23:28:39.6147619Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:28:39.6148565Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:28:39.6149733Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-08-20T23:28:39.6151281Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:28:39.6152931Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:28:39.6154134Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:28:39.6155206Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:28:39.6156227Z self._skip_by_device_name() 2024-08-20T23:28:39.6157149Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:28:39.6158386Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:28:39.6159490Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-08-20T23:28:39.6160074Z 2024-08-20T23:28:39.6160195Z model_fail_to_load 2024-08-20T23:28:43.1034037Z 2024-08-20T23:28:45.0936570Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:28:45.3503820Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:28:45.3504782Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:28:45.3506490Z using fused AdamW: True 2024-08-20T23:28:45.6841414Z 2024-08-20T23:28:45.6841821Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:28:45.6878558Z cuda eval nanogpt 2024-08-20T23:28:59.1646334Z pass 2024-08-20T23:28:59.1665447Z TIMING: code_gen:2.35686 inductor_compile:4.80974 backend_compile:9.08173 entire_frame_compile:11.91878 total_wall_time:11.91878 2024-08-20T23:28:59.1666857Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:19119 | FakeTensor.__torch_dispatch__:2052 | ProxyTorchDispatchMode.__torch_dispatch__:8279 2024-08-20T23:28:59.1668094Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:03.3695014Z 2024-08-20T23:29:05.8040071Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:05.8040553Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:29:05.8043022Z cuda eval nvidia_deeprecommender 2024-08-20T23:29:10.3006736Z pass 2024-08-20T23:29:10.3042388Z TIMING: code_gen:0.38988 inductor_compile:0.93615 backend_compile:2.92241 entire_frame_compile:3.2539 total_wall_time:3.2539 2024-08-20T23:29:10.3043754Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1366 | ProxyTorchDispatchMode.__torch_dispatch__:579 | FakeTensor.__torch_dispatch__:170 2024-08-20T23:29:10.3044851Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:14.1321193Z 2024-08-20T23:29:15.0944561Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:15.0945065Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:15.0961961Z cuda eval opacus_cifar10 2024-08-20T23:29:25.1498941Z pass 2024-08-20T23:29:25.1500193Z TIMING: code_gen:2.78999 inductor_compile:4.18025 backend_compile:6.45416 entire_frame_compile:8.67124 total_wall_time:8.67124 2024-08-20T23:29:25.1501889Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:5086 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866 2024-08-20T23:29:25.1503023Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:29.2588047Z 2024-08-20T23:29:29.9323129Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:29.9323609Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:29.9383424Z cuda eval phlippe_densenet 2024-08-20T23:29:50.4333080Z pass 2024-08-20T23:29:50.4335027Z TIMING: code_gen:4.5556 inductor_compile:7.84262 backend_compile:13.40222 entire_frame_compile:19.03245 total_wall_time:19.03245 2024-08-20T23:29:50.4336430Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:20818 | ProxyTorchDispatchMode.__torch_dispatch__:7380 | FakeTensor.__torch_dispatch__:2741 2024-08-20T23:29:50.4337556Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:54.8526421Z 2024-08-20T23:29:55.4104152Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:55.4105642Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:55.4128080Z cuda eval phlippe_resnet 2024-08-20T23:30:03.5207962Z pass 2024-08-20T23:30:03.5208902Z TIMING: code_gen:1.38456 inductor_compile:2.55857 backend_compile:4.97124 entire_frame_compile:6.86395 total_wall_time:6.86395 2024-08-20T23:30:03.5210297Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:7943 | ProxyTorchDispatchMode.__torch_dispatch__:2780 | FakeTensor.__torch_dispatch__:1018 2024-08-20T23:30:03.5211431Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:30:07.5446972Z 2024-08-20T23:30:07.8910581Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:07.8911069Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:07.8913576Z cuda eval pyhpc_equation_of_state 2024-08-20T23:30:14.5325270Z pass 2024-08-20T23:30:14.5326427Z TIMING: code_gen:1.64614 inductor_compile:3.32442 backend_compile:4.9741 entire_frame_compile:5.53344 total_wall_time:5.53344 2024-08-20T23:30:14.5328460Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:8846 | ProxyTorchDispatchMode.__torch_dispatch__:3745 | FakeTensor.__torch_dispatch__:1061 2024-08-20T23:30:14.5330036Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-08-20T23:30:18.4435347Z 2024-08-20T23:30:18.7790564Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:18.7795303Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:18.7795902Z cuda eval pyhpc_isoneutral_mixing 2024-08-20T23:30:44.2341023Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 2024-08-20T23:30:44.2342001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2546, in run_n_iterations 2024-08-20T23:30:44.2342999Z return self.model_iter_fn(mod, inputs, collect_outputs=True) 2024-08-20T23:30:44.2343991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:30:44.2344931Z return mod(*inputs) 2024-08-20T23:30:44.2346113Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 96, in forward 2024-08-20T23:30:44.2347277Z return isoneutral_pytorch.isoneutral_diffusion_pre( 2024-08-20T23:30:44.2348427Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 165, in isoneutral_diffusion_pre 2024-08-20T23:30:44.2350036Z K_11[1:-2, 2:-2, :] = sumz / (4.0 * dzt[None, None, :]) 2024-08-20T23:30:44.2350503Z 2024-08-20T23:30:44.2413080Z pass 2024-08-20T23:30:44.2413994Z TIMING: code_gen:5.24005 inductor_compile:14.53763 backend_compile:22.02367 entire_frame_compile:24.06703 total_wall_time:24.06703 2024-08-20T23:30:44.2415397Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:41362 | FakeTensor.__torch_dispatch__:5051 | ProxyTorchDispatchMode.__torch_dispatch__:14005 2024-08-20T23:30:44.2416829Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-08-20T23:30:49.1000065Z 2024-08-20T23:30:49.8725579Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:30:49.9914728Z 2024-08-20T23:30:49.9915133Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:49.9916100Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:30:49.9917024Z cuda eval pyhpc_turbulent_kinetic_energy 2024-08-20T23:30:50.0365344Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:31:18.2297818Z pass 2024-08-20T23:31:18.2357224Z TIMING: code_gen:9.95673 inductor_compile:16.12992 backend_compile:24.54484 entire_frame_compile:26.79608 total_wall_time:26.79608 2024-08-20T23:31:18.2358720Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:38601 | FakeTensor.__torch_dispatch__:2563 | ProxyTorchDispatchMode.__torch_dispatch__:14635 2024-08-20T23:31:18.2360169Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-08-20T23:31:22.9792998Z 2024-08-20T23:31:23.7898738Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:23.7899196Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:23.7911257Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-08-20T23:31:41.0754758Z pass 2024-08-20T23:31:41.0761950Z TIMING: code_gen:3.04208 inductor_compile:5.15906 backend_compile:7.16957 entire_frame_compile:8.9412 total_wall_time:8.9412 2024-08-20T23:31:41.0763349Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:8225 | ProxyTorchDispatchMode.__torch_dispatch__:3743 | FakeTensor.__torch_dispatch__:1038 2024-08-20T23:31:41.0764482Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-08-20T23:31:45.2415869Z 2024-08-20T23:31:46.4199687Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:46.4200156Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:31:46.4219413Z cuda eval pytorch_stargan 2024-08-20T23:32:11.1931560Z pass 2024-08-20T23:32:11.1942661Z TIMING: code_gen:2.35493 inductor_compile:4.53454 backend_compile:7.20752 entire_frame_compile:8.97853 total_wall_time:8.97853 2024-08-20T23:32:11.1944324Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:12390 | FakeTensor.__torch_dispatch__:1212 | ProxyTorchDispatchMode.__torch_dispatch__:4591 2024-08-20T23:32:11.1946941Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-08-20T23:32:15.3616003Z 2024-08-20T23:32:16.0659987Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:32:16.0661042Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:32:16.0680772Z cuda eval pytorch_unet 2024-08-20T23:32:38.8696456Z pass 2024-08-20T23:32:38.8719652Z TIMING: code_gen:4.19623 inductor_compile:6.35728 backend_compile:8.70956 entire_frame_compile:10.84175 total_wall_time:10.84175 2024-08-20T23:32:38.8721190Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:11341 | ProxyTorchDispatchMode.__torch_dispatch__:4337 | FakeTensor.__torch_dispatch__:1309 2024-08-20T23:32:38.8722654Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:32:42.9930372Z 2024-08-20T23:32:43.8036270Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-08-20T23:32:43.8217834Z 2024-08-20T23:32:43.8217986Z 2024-08-20T23:32:43.9221181Z 0% 0.00/230M [00:00 2024-08-20T23:51:03.0841329Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-08-20T23:51:03.0841932Z 2024-08-20T23:51:04.2269627Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:51:04.2271098Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:51:04.2272131Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] or: 2024-08-20T23:51:04.2273133Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:51:04.2274358Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] to include these operations in the captured graph. 2024-08-20T23:51:04.2275350Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:51:04.2276514Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] Graph break: from user code at: 2024-08-20T23:51:04.2278528Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104 2024-08-20T23:51:04.2280595Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets) 2024-08-20T23:51:04.2282602Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward 2024-08-20T23:51:04.2284311Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] box_features = self.box_roi_pool(features, proposals, image_shapes) 2024-08-20T23:51:04.2286264Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward 2024-08-20T23:51:04.2287821Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] self.scales, self.map_levels = _setup_scales( 2024-08-20T23:51:04.2289531Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales 2024-08-20T23:51:04.2291187Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:51:04.2292979Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in 2024-08-20T23:51:04.2294617Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:51:04.2296437Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-20T23:51:04.2298043Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-20T23:51:04.2299123Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:51:04.2299859Z W0820 23:51:04.226000 44833 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:51:46.6024766Z pass 2024-08-20T23:51:46.6038638Z TIMING: entire_frame_compile:69.61228 code_gen:13.10676 inductor_compile:22.32571 backend_compile:33.9567 total_wall_time:69.61228 2024-08-20T23:51:46.6040858Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:61102 | FakeTensor.__torch_dispatch__:6028 | ProxyTorchDispatchMode.__torch_dispatch__:18373 | attempt fast:152 | fast is_contiguous:96 | slow both tensors nontrivially broadcast:56 2024-08-20T23:51:46.6042566Z Dynamo produced 25 graphs covering 1358 ops with 16 graph breaks (2 unique) 2024-08-20T23:51:51.9421049Z 2024-08-20T23:51:54.2225948Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:51:54.2226438Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:51:54.2314162Z cuda eval yolov3 2024-08-20T23:52:19.3353744Z W0820 23:52:19.334000 45269 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:52:19.5868413Z W0820 23:52:19.586000 45269 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:52:19.8324921Z W0820 23:52:19.831000 45269 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:52:20.1415204Z W0820 23:52:20.140000 45269 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:52:20.1422495Z W0820 23:52:20.141000 45269 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:52:20.1430865Z W0820 23:52:20.142000 45269 torch/_inductor/utils.py:1391] [0/0] DeviceCopy in input program 2024-08-20T23:52:29.9308137Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg298_1). Found from : 2024-08-20T23:52:29.9309223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-20T23:52:29.9310061Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-20T23:52:29.9310931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:52:29.9311675Z return mod(*inputs) 2024-08-20T23:52:29.9312545Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-08-20T23:52:29.9313694Z return self.forward_once(x) 2024-08-20T23:52:29.9314563Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once 2024-08-20T23:52:29.9315491Z yolo_out.append(module(x, out)) 2024-08-20T23:52:29.9318005Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-08-20T23:52:29.9318921Z self.create_grids((nx, ny), p.device) 2024-08-20T23:52:29.9319822Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-08-20T23:52:29.9320744Z self.anchor_vec = self.anchor_vec.to(device) 2024-08-20T23:52:29.9321082Z 2024-08-20T23:52:32.2563231Z pass 2024-08-20T23:52:32.2676631Z TIMING: code_gen:9.52087 inductor_compile:14.90501 backend_compile:23.05492 entire_frame_compile:30.45981 total_wall_time:30.45981 2024-08-20T23:52:32.2678139Z STATS: call_* op count: 649 | FakeTensorMode.__torch_dispatch__:31336 | FakeTensor.__torch_dispatch__:3524 | ProxyTorchDispatchMode.__torch_dispatch__:12245 2024-08-20T23:52:32.2679256Z Dynamo produced 1 graphs covering 649 ops with 0 graph breaks (0 unique) 2024-08-20T23:52:34.3777119Z accuracy pass_rate=82.61% 2024-08-20T23:52:34.3781222Z calls_captured gmean=0.00x mean=662.391x 2024-08-20T23:52:34.3785184Z unique_graphs gmean=0.00x mean=1.674x 2024-08-20T23:52:34.3789282Z graph_breaks gmean=0.00x mean=0.717x 2024-08-20T23:52:34.3793348Z unique_graph_breaks gmean=0.00x mean=0.174x 2024-08-20T23:52:34.3797277Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:52:34.3801325Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:52:34.3805087Z cudagraph_skips gmean=0.00x mean=0.239x 2024-08-20T23:52:35.6144831Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv 2024-08-20T23:52:35.9582466Z llama PASS 2024-08-20T23:52:35.9588171Z llama_v2_7b_16h XFAIL 2024-08-20T23:52:35.9593402Z llava XFAIL 2024-08-20T23:52:35.9598518Z maml XFAIL 2024-08-20T23:52:35.9604004Z maml_omniglot PASS 2024-08-20T23:52:35.9609392Z mnasnet1_0 PASS 2024-08-20T23:52:35.9614815Z mobilenet_v2 PASS 2024-08-20T23:52:35.9620149Z mobilenet_v2_quantized_qat XFAIL 2024-08-20T23:52:35.9625490Z mobilenet_v3_large PASS 2024-08-20T23:52:35.9630888Z moco PASS 2024-08-20T23:52:35.9637293Z moondream XFAIL 2024-08-20T23:52:35.9642056Z nanogpt PASS 2024-08-20T23:52:35.9646817Z nvidia_deeprecommender PASS 2024-08-20T23:52:35.9651204Z opacus_cifar10 PASS 2024-08-20T23:52:35.9655824Z phlippe_densenet PASS 2024-08-20T23:52:35.9660585Z phlippe_resnet PASS 2024-08-20T23:52:35.9665023Z pyhpc_equation_of_state PASS 2024-08-20T23:52:35.9669838Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:52:35.9674183Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:52:35.9678698Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:52:35.9683219Z pytorch_stargan PASS 2024-08-20T23:52:35.9687829Z pytorch_unet PASS 2024-08-20T23:52:35.9692344Z resnet152 PASS 2024-08-20T23:52:35.9696733Z resnet18 PASS 2024-08-20T23:52:35.9701243Z resnet50 PASS 2024-08-20T23:52:35.9705722Z resnet50_quantized_qat XFAIL 2024-08-20T23:52:35.9710507Z resnext50_32x4d PASS 2024-08-20T23:52:35.9714910Z sam PASS 2024-08-20T23:52:35.9719392Z sam_fast PASS 2024-08-20T23:52:35.9723873Z shufflenet_v2_x1_0 PASS 2024-08-20T23:52:35.9728323Z soft_actor_critic PASS 2024-08-20T23:52:35.9732858Z speech_transformer PASS 2024-08-20T23:52:35.9738054Z squeezenet1_1 PASS 2024-08-20T23:52:35.9742538Z stable_diffusion_text_encoder PASS 2024-08-20T23:52:35.9747335Z stable_diffusion_unet XFAIL 2024-08-20T23:52:35.9751758Z timm_efficientnet PASS 2024-08-20T23:52:35.9756234Z timm_regnet PASS 2024-08-20T23:52:35.9760616Z timm_resnest PASS 2024-08-20T23:52:35.9765073Z timm_vision_transformer PASS 2024-08-20T23:52:35.9769503Z timm_vision_transformer_large XFAIL 2024-08-20T23:52:35.9774079Z timm_vovnet PASS 2024-08-20T23:52:35.9778531Z torch_multimodal_clip PASS 2024-08-20T23:52:35.9782966Z tts_angular PASS 2024-08-20T23:52:35.9787478Z vgg16 PASS 2024-08-20T23:52:35.9792054Z vision_maskrcnn PASS 2024-08-20T23:52:35.9796445Z yolov3 PASS 2024-08-20T23:52:36.0322536Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv 2024-08-20T23:52:36.3703740Z llama PASS 2024-08-20T23:52:36.3708417Z llama_v2_7b_16h PASS 2024-08-20T23:52:36.3712932Z llava PASS 2024-08-20T23:52:36.3717305Z maml PASS 2024-08-20T23:52:36.3721509Z maml_omniglot PASS 2024-08-20T23:52:36.3725843Z mnasnet1_0 PASS 2024-08-20T23:52:36.3730475Z mobilenet_v2 PASS 2024-08-20T23:52:36.3735295Z mobilenet_v2_quantized_qat PASS 2024-08-20T23:52:36.3739729Z mobilenet_v3_large PASS 2024-08-20T23:52:36.3743975Z moco PASS 2024-08-20T23:52:36.3748414Z moondream PASS 2024-08-20T23:52:36.3752868Z nanogpt PASS 2024-08-20T23:52:36.3757110Z nvidia_deeprecommender PASS 2024-08-20T23:52:36.3761484Z opacus_cifar10 PASS 2024-08-20T23:52:36.3765701Z phlippe_densenet PASS 2024-08-20T23:52:36.3770076Z phlippe_resnet PASS 2024-08-20T23:52:36.3774255Z pyhpc_equation_of_state PASS 2024-08-20T23:52:36.3778577Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:52:36.3782902Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:52:36.3787332Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:52:36.3791909Z pytorch_stargan PASS 2024-08-20T23:52:36.3796059Z pytorch_unet PASS 2024-08-20T23:52:36.3800351Z resnet152 PASS 2024-08-20T23:52:36.3804589Z resnet18 PASS 2024-08-20T23:52:36.3808982Z resnet50 PASS 2024-08-20T23:52:36.3813293Z resnet50_quantized_qat PASS 2024-08-20T23:52:36.3817527Z resnext50_32x4d PASS 2024-08-20T23:52:36.3821895Z sam PASS 2024-08-20T23:52:36.3826214Z sam_fast PASS 2024-08-20T23:52:36.3830783Z shufflenet_v2_x1_0 PASS 2024-08-20T23:52:36.3835138Z soft_actor_critic PASS 2024-08-20T23:52:36.3839979Z speech_transformer PASS 2024-08-20T23:52:36.3844222Z squeezenet1_1 PASS 2024-08-20T23:52:36.3848563Z stable_diffusion_text_encoder PASS 2024-08-20T23:52:36.3852838Z stable_diffusion_unet PASS 2024-08-20T23:52:36.3857086Z timm_efficientnet PASS 2024-08-20T23:52:36.3861416Z timm_regnet PASS 2024-08-20T23:52:36.3865771Z timm_resnest PASS 2024-08-20T23:52:36.3870255Z timm_vision_transformer PASS 2024-08-20T23:52:36.3874542Z timm_vision_transformer_large PASS 2024-08-20T23:52:36.3878878Z timm_vovnet PASS 2024-08-20T23:52:36.3883159Z torch_multimodal_clip PASS 2024-08-20T23:52:36.3887461Z tts_angular PASS 2024-08-20T23:52:36.3891949Z vgg16 PASS 2024-08-20T23:52:36.3896111Z vision_maskrcnn PASS 2024-08-20T23:52:36.3900447Z yolov3 PASS 2024-08-20T23:52:36.4416771Z + test_single_dynamo_benchmark training torchbench 1 --training --amp 2024-08-20T23:52:36.4420219Z ++ pwd 2024-08-20T23:52:36.4423755Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:52:36.4424480Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:52:36.4448325Z + local name=training 2024-08-20T23:52:36.4449017Z + shift 2024-08-20T23:52:36.4449401Z + local suite=torchbench 2024-08-20T23:52:36.4449869Z + shift 2024-08-20T23:52:36.4450223Z + local shard_id=1 2024-08-20T23:52:36.4450623Z + shift 2024-08-20T23:52:36.4450914Z + partition_flags=() 2024-08-20T23:52:36.4451273Z + local partition_flags 2024-08-20T23:52:36.4451758Z + [[ -n 2 ]] 2024-08-20T23:52:36.4452071Z + [[ -n 1 ]] 2024-08-20T23:52:36.4452765Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:52:36.4453523Z + [[ inductor_torchbench == *perf_compare* ]] 2024-08-20T23:52:36.4454056Z + [[ inductor_torchbench == *perf* ]] 2024-08-20T23:52:36.4454626Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:52:36.4455227Z + [[ inductor_torchbench == *_avx2* ]] 2024-08-20T23:52:36.4455829Z + [[ inductor_torchbench == *_avx512* ]] 2024-08-20T23:52:36.4457608Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2024-08-20T23:52:42.1716502Z 2024-08-20T23:52:42.4728386Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:42.4729412Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:42.4730314Z cuda train llava 2024-08-20T23:52:42.4734227Z Traceback (most recent call last): 2024-08-20T23:52:42.4735174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:52:42.4736081Z ) = runner.load_model( 2024-08-20T23:52:42.4736776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model 2024-08-20T23:52:42.4737509Z benchmark = benchmark_cls( 2024-08-20T23:52:42.4738259Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:52:42.4739064Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:52:42.4739929Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:52:42.4741058Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:52:42.4742322Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:52:42.4743792Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:52:42.4744799Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:52:42.4745570Z self._skip_by_device_name() 2024-08-20T23:52:42.4746387Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:52:42.4747814Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:52:42.4748915Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:52:42.4749491Z 2024-08-20T23:52:42.4749618Z model_fail_to_load 2024-08-20T23:52:45.9345550Z 2024-08-20T23:52:46.5722358Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:46.5722823Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:46.5723292Z cuda train maml_omniglot 2024-08-20T23:52:55.7009081Z W0820 23:52:55.699000 45717 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:53:01.3956546Z pass 2024-08-20T23:53:01.3957565Z TIMING: entire_frame_compile:9.43552 code_gen:7.41837 inductor_compile:8.8733 backend_compile:8.27724 total_wall_time:12.9491 2024-08-20T23:53:01.3958958Z STATS: call_* op count: 77 | FakeTensorMode.__torch_dispatch__:4092 | ProxyTorchDispatchMode.__torch_dispatch__:1526 | FakeTensor.__torch_dispatch__:1147 2024-08-20T23:53:01.3960085Z Dynamo produced 3 graphs covering 77 ops with 7 graph breaks (5 unique) 2024-08-20T23:53:05.5183221Z 2024-08-20T23:53:08.0033734Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:08.0034344Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:53:08.0036797Z cuda train mnasnet1_0 2024-08-20T23:53:44.7148554Z W0820 23:53:44.714000 46054 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:54:24.6290397Z pass 2024-08-20T23:54:24.6319596Z TIMING: entire_frame_compile:57.04779 code_gen:24.50307 inductor_compile:37.72382 backend_compile:42.60282 total_wall_time:68.57025 2024-08-20T23:54:24.6321308Z STATS: call_* op count: 647 | FakeTensorMode.__torch_dispatch__:41893 | ProxyTorchDispatchMode.__torch_dispatch__:17470 | FakeTensor.__torch_dispatch__:13531 2024-08-20T23:54:24.6322598Z Dynamo produced 3 graphs covering 647 ops with 7 graph breaks (5 unique) 2024-08-20T23:54:31.2529946Z 2024-08-20T23:54:33.2632367Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:54:33.2632934Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:54:33.2636873Z cuda train mobilenet_v2 2024-08-20T23:55:10.7363320Z pass 2024-08-20T23:55:10.7415077Z TIMING: entire_frame_compile:19.89548 code_gen:10.12914 inductor_compile:17.45871 backend_compile:12.74425 total_wall_time:32.25768 2024-08-20T23:55:10.7417243Z STATS: call_* op count: 157 | FakeTensorMode.__torch_dispatch__:26530 | FakeTensor.__torch_dispatch__:6732 | ProxyTorchDispatchMode.__torch_dispatch__:10376 2024-08-20T23:55:10.7418370Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-08-20T23:55:15.7717489Z 2024-08-20T23:55:17.7348582Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:17.7349193Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:55:17.7351523Z cuda train mobilenet_v2_quantized_qat 2024-08-20T23:55:17.7352136Z Traceback (most recent call last): 2024-08-20T23:55:17.7352962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:55:17.7353729Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:55:17.7354592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-20T23:55:17.7357236Z pred = mod(*cloned_inputs) 2024-08-20T23:55:17.7358927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-20T23:55:17.7359831Z return self._wrapped_call(self, *args, **kwargs) 2024-08-20T23:55:17.7360790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-20T23:55:17.7361578Z raise e 2024-08-20T23:55:17.7362587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-20T23:55:17.7363864Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:55:17.7365356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:55:17.7366276Z return self._call_impl(*args, **kwargs) 2024-08-20T23:55:17.7367224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:55:17.7368081Z return forward_call(*args, **kwargs) 2024-08-20T23:55:17.7368591Z File ".3", line 207, in forward 2024-08-20T23:55:17.7369672Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-08-20T23:55:17.7370962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:55:17.7371860Z return self._call_impl(*args, **kwargs) 2024-08-20T23:55:17.7372804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:55:17.7373640Z return forward_call(*args, **kwargs) 2024-08-20T23:55:17.7374621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-20T23:55:17.7375542Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-20T23:55:17.7376095Z RuntimeError: expected scalar type Float but found Half 2024-08-20T23:55:17.7376478Z 2024-08-20T23:55:17.7376775Z The above exception was the direct cause of the following exception: 2024-08-20T23:55:17.7377229Z 2024-08-20T23:55:17.7377380Z Traceback (most recent call last): 2024-08-20T23:55:17.7378032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:55:17.7378694Z ) = runner.load_model( 2024-08-20T23:55:17.7379342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:55:17.7380112Z self.validate_model(model, example_inputs) 2024-08-20T23:55:17.7380867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:55:17.7381631Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:55:17.7382092Z RuntimeError: Eager run failed 2024-08-20T23:55:17.7382352Z 2024-08-20T23:55:17.7382467Z eager_fail_to_run 2024-08-20T23:55:21.3596749Z 2024-08-20T23:55:23.7701581Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:23.7702288Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:55:23.7702893Z cuda train mobilenet_v3_large 2024-08-20T23:56:07.0106050Z W0820 23:56:07.009000 47876 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:56:51.3789776Z E0820 23:56:51.377000 47876 torch/_dynamo/utils.py:1688] RMSE (res-fp64): 3.22718, (ref-fp64): 1.00780 and shape=torch.Size([16, 1, 3, 3]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000, use_larger_multiplier_for_smaller_tensor: 0 2024-08-20T23:56:51.3792802Z E0820 23:56:51.378000 47876 torch/_dynamo/utils.py:1580] Accuracy failed for key name features.1.block.0.0.weight.grad 2024-08-20T23:56:51.3816046Z pass 2024-08-20T23:56:51.3856452Z TIMING: entire_frame_compile:64.66046 code_gen:28.49159 inductor_compile:44.24972 backend_compile:48.19854 total_wall_time:79.40025 2024-08-20T23:56:51.3858488Z STATS: call_* op count: 730 | FakeTensorMode.__torch_dispatch__:46605 | FakeTensor.__torch_dispatch__:16267 | ProxyTorchDispatchMode.__torch_dispatch__:19401 2024-08-20T23:56:51.3860419Z Dynamo produced 3 graphs covering 730 ops with 7 graph breaks (5 unique) 2024-08-20T23:56:58.5489202Z 2024-08-20T23:57:00.8784768Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:57:00.8785640Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:57:00.8786483Z cuda train moco 2024-08-20T23:57:06.2422652Z [rank0]:W0820 23:57:06.241000 49070 torch/_logging/_internal.py:1058] [3/0] Profiler function will be ignored 2024-08-20T23:57:14.3301885Z [rank0]:W0820 23:57:14.329000 49070 torch/_dynamo/backends/distributed.py:89] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:57:14.3548161Z [rank0]:W0820 23:57:14.354000 49070 torch/_dynamo/backends/distributed.py:106] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:57:14.3552556Z [rank0]:W0820 23:57:14.354000 49070 torch/_dynamo/backends/distributed.py:106] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:57:14.3554077Z [rank0]:W0820 23:57:14.354000 49070 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:57:14.3555562Z [rank0]:W0820 23:57:14.354000 49070 torch/_dynamo/backends/distributed.py:106] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:57:14.3557040Z [rank0]:W0820 23:57:14.354000 49070 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ 0 │ 480 │ 94032128 │ 2024-08-20T23:57:14.3558466Z [rank0]:W0820 23:57:14.354000 49070 torch/_dynamo/backends/distributed.py:106] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:57:40.4868613Z skipping cudagraphs due to mutated inputs (161 instances) 2024-08-20T23:57:40.5895867Z [rank0]:W0820 23:57:40.588000 49070 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-20T23:57:40.6018276Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-08-20T23:57:40.6019745Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-08-20T23:57:40.6021111Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-08-20T23:57:40.6021843Z 2024-08-20T23:57:43.5004796Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:57:43.5006430Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:57:43.5007947Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] or: 2024-08-20T23:57:43.5009042Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:57:43.5010732Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] to include these operations in the captured graph. 2024-08-20T23:57:43.5012107Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:57:43.5013271Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break: from user code at: 2024-08-20T23:57:43.5015194Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:57:43.5016937Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] self._dequeue_and_enqueue(k) 2024-08-20T23:57:43.5019256Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:57:43.5020809Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] return func(*args, **kwargs) 2024-08-20T23:57:43.5022460Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:57:43.5024093Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] ptr = int(self.queue_ptr) 2024-08-20T23:57:43.5025059Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:57:43.5025883Z [rank0]:W0820 23:57:43.499000 49070 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:58:18.5268827Z pass 2024-08-20T23:58:18.5604236Z TIMING: entire_frame_compile:52.00449 inductor_compile:32.98936 backend_compile:35.12533 code_gen:20.39938 total_wall_time:66.48712 2024-08-20T23:58:18.5605876Z STATS: call_* op count: 889 | FakeTensorMode.__torch_dispatch__:57159 | ProxyTorchDispatchMode.__torch_dispatch__:20320 | FakeTensor.__torch_dispatch__:10831 2024-08-20T23:58:18.5607357Z Dynamo produced 10 graphs covering 889 ops with 11 graph breaks (8 unique) 2024-08-20T23:58:20.1078249Z [rank0]:[W820 23:58:20.563863473 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:58:25.4808959Z 2024-08-20T23:58:27.4569601Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:58:27.7208612Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:58:27.7209661Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:58:27.7210211Z using fused AdamW: True 2024-08-20T23:58:28.1631284Z 2024-08-20T23:58:28.1631643Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:58:28.1632102Z cuda train nanogpt 2024-08-20T23:58:54.3158915Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:58:54.3160312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-20T23:58:54.3161470Z pred = mod(*cloned_inputs) 2024-08-20T23:58:54.3162568Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 228, in forward 2024-08-20T23:58:54.3163737Z tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) 2024-08-20T23:58:54.3164247Z 2024-08-20T23:58:54.3787485Z W0820 23:58:54.377000 50147 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:59:28.2785572Z pass 2024-08-20T23:59:28.3363605Z TIMING: entire_frame_compile:47.41399 code_gen:20.55098 inductor_compile:32.55362 backend_compile:38.37811 total_wall_time:55.77176 2024-08-20T23:59:28.3365126Z STATS: call_* op count: 858 | FakeTensorMode.__torch_dispatch__:43039 | FakeTensor.__torch_dispatch__:12707 | ProxyTorchDispatchMode.__torch_dispatch__:19297 2024-08-20T23:59:28.3366261Z Dynamo produced 3 graphs covering 858 ops with 7 graph breaks (5 unique) 2024-08-20T23:59:34.6179741Z 2024-08-20T23:59:37.1154911Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:37.1155548Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:59:37.1157555Z cuda train nvidia_deeprecommender 2024-08-20T23:59:45.4308378Z W0820 23:59:45.429000 50517 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:59:50.3446900Z pass 2024-08-20T23:59:50.3737177Z TIMING: entire_frame_compile:9.74888 code_gen:3.73554 inductor_compile:4.90546 backend_compile:8.79415 total_wall_time:10.74228 2024-08-20T23:59:50.3738579Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:3676 | ProxyTorchDispatchMode.__torch_dispatch__:1501 | FakeTensor.__torch_dispatch__:1172 2024-08-20T23:59:50.3740059Z Dynamo produced 3 graphs covering 70 ops with 7 graph breaks (5 unique) 2024-08-20T23:59:54.3969548Z 2024-08-20T23:59:55.3701762Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:55.3702373Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:55.3702857Z cuda train opacus_cifar10 2024-08-20T23:59:55.3716364Z Traceback (most recent call last): 2024-08-20T23:59:55.3717347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:59:55.3718219Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:59:55.3720513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-20T23:59:55.3721708Z self.grad_scaler.scale(loss).backward() 2024-08-20T23:59:55.3722830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-20T23:59:55.3723596Z torch.autograd.backward( 2024-08-20T23:59:55.3724547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-20T23:59:55.3725501Z _engine_run_backward( 2024-08-20T23:59:55.3726612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-20T23:59:55.3727791Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-20T23:59:55.3729015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 98, in __call__ 2024-08-20T23:59:55.3729868Z return self.hook(module, *args, **kwargs) 2024-08-20T23:59:55.3731028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 328, in capture_backprops_hook 2024-08-20T23:59:55.3732397Z activations, backprops = self.rearrange_grad_samples( 2024-08-20T23:59:55.3734032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 384, in rearrange_grad_samples 2024-08-20T23:59:55.3735228Z raise ValueError( 2024-08-20T23:59:55.3736282Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-08-20T23:59:55.3737009Z 2024-08-20T23:59:55.3737307Z The above exception was the direct cause of the following exception: 2024-08-20T23:59:55.3737761Z 2024-08-20T23:59:55.3737909Z Traceback (most recent call last): 2024-08-20T23:59:55.3738580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:59:55.3739240Z ) = runner.load_model( 2024-08-20T23:59:55.3739908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:59:55.3740676Z self.validate_model(model, example_inputs) 2024-08-20T23:59:55.3741437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:59:55.3742205Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:59:55.3742684Z RuntimeError: Eager run failed 2024-08-20T23:59:55.3742948Z 2024-08-20T23:59:55.3743072Z eager_fail_to_run 2024-08-20T23:59:58.9677193Z 2024-08-20T23:59:59.7983972Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:59.7984591Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:59:59.7985103Z cuda train phlippe_densenet 2024-08-21T00:00:34.0505482Z pass 2024-08-21T00:00:34.0506846Z TIMING: entire_frame_compile:21.94529 code_gen:10.67912 inductor_compile:16.15607 backend_compile:14.7039 total_wall_time:31.69012 2024-08-21T00:00:34.0508365Z STATS: call_* op count: 190 | FakeTensorMode.__torch_dispatch__:26612 | ProxyTorchDispatchMode.__torch_dispatch__:11247 | FakeTensor.__torch_dispatch__:5672 2024-08-21T00:00:34.0509510Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-08-21T00:00:38.8956927Z 2024-08-21T00:00:39.6423395Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:39.6423886Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:39.6424355Z cuda train phlippe_resnet 2024-08-21T00:00:52.6518320Z pass 2024-08-21T00:00:52.6519330Z TIMING: entire_frame_compile:8.22631 code_gen:2.97546 inductor_compile:5.08413 backend_compile:5.7271 total_wall_time:11.16787 2024-08-21T00:00:52.6520711Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:10207 | ProxyTorchDispatchMode.__torch_dispatch__:4228 | FakeTensor.__torch_dispatch__:2127 2024-08-21T00:00:52.6521857Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-08-21T00:00:56.8592670Z 2024-08-21T00:00:58.1208539Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:58.1209046Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:00:58.1209501Z cuda train pytorch_CycleGAN_and_pix2pix 2024-08-21T00:01:34.8123450Z pass 2024-08-21T00:01:34.8214348Z TIMING: entire_frame_compile:11.48402 code_gen:10.57801 inductor_compile:16.4627 backend_compile:9.31807 total_wall_time:23.15861 2024-08-21T00:01:34.8217158Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:13910 | ProxyTorchDispatchMode.__torch_dispatch__:7137 | FakeTensor.__torch_dispatch__:3594 2024-08-21T00:01:34.8219386Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-08-21T00:01:39.4973525Z 2024-08-21T00:01:41.0217676Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:01:41.0218154Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:01:41.0218603Z cuda train pytorch_stargan 2024-08-21T00:02:21.0949104Z pass 2024-08-21T00:02:21.1025880Z TIMING: entire_frame_compile:9.22067 code_gen:3.78071 inductor_compile:6.90001 backend_compile:6.85751 total_wall_time:12.44183 2024-08-21T00:02:21.1027596Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:13106 | FakeTensor.__torch_dispatch__:2199 | ProxyTorchDispatchMode.__torch_dispatch__:5359 2024-08-21T00:02:21.1028919Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-21T00:02:25.4729776Z 2024-08-21T00:02:26.4246936Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:02:26.4247407Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:02:26.4247858Z cuda train pytorch_unet 2024-08-21T00:03:18.9311504Z W0821 00:03:18.930000 52607 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:03:38.8083378Z pass_due_to_skip 2024-08-21T00:03:38.8177394Z TIMING: entire_frame_compile:29.78553 code_gen:20.38022 inductor_compile:27.38135 backend_compile:23.7757 total_wall_time:40.66436 2024-08-21T00:03:38.8179818Z STATS: call_* op count: 314 | FakeTensorMode.__torch_dispatch__:21318 | ProxyTorchDispatchMode.__torch_dispatch__:9051 | FakeTensor.__torch_dispatch__:6599 2024-08-21T00:03:38.8181905Z Dynamo produced 3 graphs covering 314 ops with 7 graph breaks (5 unique) 2024-08-21T00:03:44.0195555Z 2024-08-21T00:03:46.4201555Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:03:46.4202037Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:03:46.4202479Z cuda train resnet152 2024-08-21T00:05:11.9941604Z W0821 00:05:11.993000 53766 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:07:10.8881175Z pass 2024-08-21T00:07:10.9270859Z TIMING: entire_frame_compile:162.55312 code_gen:61.39672 inductor_compile:98.27843 backend_compile:120.25199 total_wall_time:187.79386 2024-08-21T00:07:10.9272448Z STATS: call_* op count: 1937 | FakeTensorMode.__torch_dispatch__:123277 | ProxyTorchDispatchMode.__torch_dispatch__:53198 | FakeTensor.__torch_dispatch__:39779 2024-08-21T00:07:10.9275188Z Dynamo produced 3 graphs covering 1937 ops with 7 graph breaks (5 unique) 2024-08-21T00:07:23.1316356Z 2024-08-21T00:07:24.4151848Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:07:24.4152521Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:07:24.4153129Z cuda train resnet18 2024-08-21T00:07:42.2480187Z pass 2024-08-21T00:07:42.2552002Z TIMING: entire_frame_compile:9.0036 code_gen:5.73308 inductor_compile:8.316 backend_compile:6.6761 total_wall_time:14.5406 2024-08-21T00:07:42.2553395Z STATS: call_* op count: 73 | FakeTensorMode.__torch_dispatch__:10423 | ProxyTorchDispatchMode.__torch_dispatch__:4224 | FakeTensor.__torch_dispatch__:2162 2024-08-21T00:07:42.2555128Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-08-21T00:07:46.6449024Z 2024-08-21T00:07:48.4828605Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:07:48.4829100Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:07:48.4829558Z cuda train resnet50 2024-08-21T00:08:25.0132559Z pass 2024-08-21T00:08:25.0298714Z TIMING: entire_frame_compile:19.38086 code_gen:9.72158 inductor_compile:15.5056 backend_compile:12.65323 total_wall_time:29.92038 2024-08-21T00:08:25.0300240Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:26313 | ProxyTorchDispatchMode.__torch_dispatch__:11011 | FakeTensor.__torch_dispatch__:5605 2024-08-21T00:08:25.0301524Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-08-21T00:08:30.0266265Z 2024-08-21T00:08:32.1835468Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:08:32.1836086Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:08:32.1836723Z cuda train resnet50_quantized_qat 2024-08-21T00:08:32.1837396Z Traceback (most recent call last): 2024-08-21T00:08:32.1839541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-21T00:08:32.1840621Z self.model_iter_fn(model, example_inputs) 2024-08-21T00:08:32.1841672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-21T00:08:32.1842563Z pred = mod(*cloned_inputs) 2024-08-21T00:08:32.1843934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-21T00:08:32.1844948Z return self._wrapped_call(self, *args, **kwargs) 2024-08-21T00:08:32.1846202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-21T00:08:32.1847190Z raise e 2024-08-21T00:08:32.1847977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-21T00:08:32.1848952Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-21T00:08:32.1850411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:08:32.1851336Z return self._call_impl(*args, **kwargs) 2024-08-21T00:08:32.1852364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:08:32.1853442Z return forward_call(*args, **kwargs) 2024-08-21T00:08:32.1854078Z File ".3", line 167, in forward 2024-08-21T00:08:32.1854941Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-08-21T00:08:32.1856379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:08:32.1857285Z return self._call_impl(*args, **kwargs) 2024-08-21T00:08:32.1858233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:08:32.1859068Z return forward_call(*args, **kwargs) 2024-08-21T00:08:32.1860380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-21T00:08:32.1861292Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-21T00:08:32.1861839Z RuntimeError: expected scalar type Float but found Half 2024-08-21T00:08:32.1862210Z 2024-08-21T00:08:32.1862512Z The above exception was the direct cause of the following exception: 2024-08-21T00:08:32.1862956Z 2024-08-21T00:08:32.1863107Z Traceback (most recent call last): 2024-08-21T00:08:32.1863750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:08:32.1864410Z ) = runner.load_model( 2024-08-21T00:08:32.1865073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-21T00:08:32.1865832Z self.validate_model(model, example_inputs) 2024-08-21T00:08:32.1866587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-21T00:08:32.1867363Z raise RuntimeError("Eager run failed") from e 2024-08-21T00:08:32.1867958Z RuntimeError: Eager run failed 2024-08-21T00:08:32.1868211Z 2024-08-21T00:08:32.1868502Z eager_fail_to_run 2024-08-21T00:08:35.8177798Z 2024-08-21T00:08:37.8039903Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:08:37.8040413Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:08:37.8040856Z cuda train resnext50_32x4d 2024-08-21T00:09:14.5844000Z W0821 00:09:14.583000 54561 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:09:54.7701585Z pass 2024-08-21T00:09:54.7821951Z TIMING: entire_frame_compile:56.95711 code_gen:23.29619 inductor_compile:36.02009 backend_compile:42.62446 total_wall_time:67.18142 2024-08-21T00:09:54.7823366Z STATS: call_* op count: 679 | FakeTensorMode.__torch_dispatch__:42566 | ProxyTorchDispatchMode.__torch_dispatch__:18280 | FakeTensor.__torch_dispatch__:13739 2024-08-21T00:09:54.7824537Z Dynamo produced 3 graphs covering 679 ops with 7 graph breaks (5 unique) 2024-08-21T00:10:01.3468480Z 2024-08-21T00:10:09.0052437Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:10:09.0052936Z loading model: 0it [00:07, ?it/s] 2024-08-21T00:10:09.0053384Z cuda train sam 2024-08-21T00:10:09.0063095Z Traceback (most recent call last): 2024-08-21T00:10:09.0064060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-21T00:10:09.0064874Z self.model_iter_fn(model, example_inputs) 2024-08-21T00:10:09.0065959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-21T00:10:09.0067254Z self.grad_scaler.scale(loss).backward() 2024-08-21T00:10:09.0068376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-21T00:10:09.0069118Z torch.autograd.backward( 2024-08-21T00:10:09.0070283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-21T00:10:09.0071152Z _engine_run_backward( 2024-08-21T00:10:09.0072270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-21T00:10:09.0073752Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-21T00:10:09.0074913Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-08-21T00:10:09.0075515Z 2024-08-21T00:10:09.0075898Z The above exception was the direct cause of the following exception: 2024-08-21T00:10:09.0076517Z 2024-08-21T00:10:09.0076700Z Traceback (most recent call last): 2024-08-21T00:10:09.0077588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:10:09.0078253Z ) = runner.load_model( 2024-08-21T00:10:09.0078918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-21T00:10:09.0080022Z self.validate_model(model, example_inputs) 2024-08-21T00:10:09.0080789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-21T00:10:09.0081563Z raise RuntimeError("Eager run failed") from e 2024-08-21T00:10:09.0082044Z RuntimeError: Eager run failed 2024-08-21T00:10:09.0082305Z 2024-08-21T00:10:09.0082421Z eager_fail_to_run 2024-08-21T00:10:12.6575070Z 2024-08-21T00:10:13.9665159Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:10:13.9665774Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:10:13.9666324Z cuda train shufflenet_v2_x1_0 2024-08-21T00:10:57.9846291Z pass 2024-08-21T00:10:57.9858247Z TIMING: entire_frame_compile:23.16396 code_gen:16.35422 inductor_compile:24.05188 backend_compile:16.07922 total_wall_time:40.13694 2024-08-21T00:10:57.9859865Z STATS: call_* op count: 275 | FakeTensorMode.__torch_dispatch__:30039 | ProxyTorchDispatchMode.__torch_dispatch__:11975 | FakeTensor.__torch_dispatch__:6677 2024-08-21T00:10:57.9861021Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique) 2024-08-21T00:11:03.2917464Z 2024-08-21T00:11:04.1642914Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:11:04.1643622Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:11:04.1644286Z cuda train soft_actor_critic 2024-08-21T00:11:09.2464899Z W0821 00:11:09.245000 55871 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:11:11.8592727Z pass 2024-08-21T00:11:11.8600442Z TIMING: entire_frame_compile:4.97831 code_gen:2.00425 inductor_compile:2.83212 backend_compile:4.35424 total_wall_time:5.72823 2024-08-21T00:11:11.8601932Z STATS: call_* op count: 56 | FakeTensorMode.__torch_dispatch__:2288 | ProxyTorchDispatchMode.__torch_dispatch__:894 | FakeTensor.__torch_dispatch__:556 2024-08-21T00:11:11.8603046Z Dynamo produced 4 graphs covering 56 ops with 6 graph breaks (5 unique) 2024-08-21T00:11:15.8405339Z 2024-08-21T00:11:17.1389678Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:11:17.1390168Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:11:17.1390649Z cuda train speech_transformer 2024-08-21T00:11:32.5237916Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_14). Found from : 2024-08-21T00:11:32.5239820Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 58, in torch_dynamo_resume_in_forward_at_50 2024-08-21T00:11:32.5241044Z enc_output, enc_slf_attn = enc_layer( 2024-08-21T00:11:32.5242126Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 84, in forward 2024-08-21T00:11:32.5243216Z enc_output, enc_slf_attn = self.slf_attn( 2024-08-21T00:11:32.5244431Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward 2024-08-21T00:11:32.5245674Z output, attn = self.attention(q, k, v, mask=mask) 2024-08-21T00:11:32.5246957Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward 2024-08-21T00:11:32.5248099Z attn = attn / self.temperature 2024-08-21T00:11:32.5248389Z 2024-08-21T00:11:55.0091623Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 2024-08-21T00:11:55.0094165Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 105, in torch_dynamo_resume_in_forward_at_96 2024-08-21T00:11:55.0096238Z dec_output, dec_slf_attn, dec_enc_attn = dec_layer( 2024-08-21T00:11:55.0097875Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 243, in forward 2024-08-21T00:11:55.0099966Z dec_output, dec_slf_attn = self.slf_attn( 2024-08-21T00:11:55.0101709Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward 2024-08-21T00:11:55.0103495Z output, attn = self.attention(q, k, v, mask=mask) 2024-08-21T00:11:55.0105313Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward 2024-08-21T00:11:55.0106997Z attn = attn / self.temperature 2024-08-21T00:11:55.0107600Z 2024-08-21T00:12:13.7070182Z pass 2024-08-21T00:12:13.7446172Z TIMING: entire_frame_compile:35.64832 inductor_compile:29.23349 backend_compile:30.9961 code_gen:19.0812 total_wall_time:50.60906 2024-08-21T00:12:13.7447606Z STATS: call_* op count: 771 | FakeTensorMode.__torch_dispatch__:50818 | ProxyTorchDispatchMode.__torch_dispatch__:25323 | FakeTensor.__torch_dispatch__:7079 2024-08-21T00:12:13.7448760Z Dynamo produced 10 graphs covering 771 ops with 16 graph breaks (7 unique) 2024-08-21T00:12:19.4356383Z 2024-08-21T00:12:20.5752396Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:20.5753362Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:12:20.5753823Z cuda train squeezenet1_1 2024-08-21T00:12:39.9263988Z pass 2024-08-21T00:12:39.9284916Z TIMING: entire_frame_compile:6.70038 code_gen:11.41862 inductor_compile:13.38288 backend_compile:5.1715 total_wall_time:16.50242 2024-08-21T00:12:39.9286334Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:6362 | ProxyTorchDispatchMode.__torch_dispatch__:2530 | FakeTensor.__torch_dispatch__:971 2024-08-21T00:12:39.9287458Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-08-21T00:12:44.2877625Z 2024-08-21T00:12:45.5478409Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:45.5478769Z 2024-08-21T00:12:46.6079117Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-08-21T00:15:51.4405604Z pass 2024-08-21T00:15:51.4461484Z TIMING: entire_frame_compile:77.41928 code_gen:33.63779 inductor_compile:52.14806 backend_compile:57.96948 total_wall_time:93.39211 2024-08-21T00:15:51.4462917Z STATS: call_* op count: 899 | FakeTensorMode.__torch_dispatch__:56685 | FakeTensor.__torch_dispatch__:19957 | ProxyTorchDispatchMode.__torch_dispatch__:24017 2024-08-21T00:15:51.4464057Z Dynamo produced 3 graphs covering 899 ops with 7 graph breaks (5 unique) 2024-08-21T00:15:59.5324230Z 2024-08-21T00:16:02.3463535Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:16:02.3464139Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:16:02.3464683Z cuda train timm_regnet 2024-08-21T00:16:57.9758102Z pass 2024-08-21T00:16:58.0176800Z TIMING: entire_frame_compile:29.8872 code_gen:11.96653 inductor_compile:21.00738 backend_compile:19.00318 total_wall_time:42.97382 2024-08-21T00:16:58.0178379Z STATS: call_* op count: 323 | FakeTensorMode.__torch_dispatch__:39633 | ProxyTorchDispatchMode.__torch_dispatch__:17103 | FakeTensor.__torch_dispatch__:9034 2024-08-21T00:16:58.0179523Z Dynamo produced 2 graphs covering 323 ops with 6 graph breaks (5 unique) 2024-08-21T00:17:03.7364324Z 2024-08-21T00:17:05.7614372Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:17:05.7614988Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:17:05.7619100Z cuda train timm_resnest 2024-08-21T00:17:38.0519289Z W0821 00:17:38.050000 59050 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:18:00.3303668Z pass 2024-08-21T00:18:00.3374782Z TIMING: entire_frame_compile:34.03384 code_gen:23.5668 inductor_compile:31.2494 backend_compile:26.69136 total_wall_time:47.67308 2024-08-21T00:18:00.3377648Z STATS: call_* op count: 401 | FakeTensorMode.__torch_dispatch__:23381 | ProxyTorchDispatchMode.__torch_dispatch__:9728 | FakeTensor.__torch_dispatch__:7416 2024-08-21T00:18:00.3378785Z Dynamo produced 3 graphs covering 401 ops with 7 graph breaks (5 unique) 2024-08-21T00:18:05.9816837Z 2024-08-21T00:18:07.5605952Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:07.5606675Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:18:07.5607385Z cuda train timm_vision_transformer 2024-08-21T00:18:37.1347798Z pass 2024-08-21T00:18:37.1521577Z TIMING: entire_frame_compile:17.71386 code_gen:8.67465 inductor_compile:13.82971 backend_compile:14.19126 total_wall_time:26.28914 2024-08-21T00:18:37.1523136Z STATS: call_* op count: 267 | FakeTensorMode.__torch_dispatch__:24115 | FakeTensor.__torch_dispatch__:5275 | ProxyTorchDispatchMode.__torch_dispatch__:10943 2024-08-21T00:18:37.1524936Z Dynamo produced 2 graphs covering 267 ops with 6 graph breaks (5 unique) 2024-08-21T00:18:42.2139593Z 2024-08-21T00:18:54.9184446Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:54.9184914Z loading model: 0it [00:12, ?it/s] 2024-08-21T00:18:54.9185368Z cuda train timm_vision_transformer_large 2024-08-21T00:18:54.9188519Z pass_due_to_skip 2024-08-21T00:18:55.1870050Z TIMING: 2024-08-21T00:18:55.1870469Z STATS: call_* op count: 0 2024-08-21T00:18:55.1871191Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:18:58.8954308Z 2024-08-21T00:19:00.9649803Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:19:00.9650406Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:19:00.9650888Z cuda train timm_vovnet 2024-08-21T00:19:40.7977490Z pass 2024-08-21T00:19:40.8161677Z TIMING: entire_frame_compile:17.43025 code_gen:15.03307 inductor_compile:20.2162 backend_compile:11.56937 total_wall_time:31.70434 2024-08-21T00:19:40.8163734Z STATS: call_* op count: 134 | FakeTensorMode.__torch_dispatch__:20162 | ProxyTorchDispatchMode.__torch_dispatch__:8526 | FakeTensor.__torch_dispatch__:4486 2024-08-21T00:19:40.8164883Z Dynamo produced 2 graphs covering 134 ops with 6 graph breaks (5 unique) 2024-08-21T00:19:45.8249267Z 2024-08-21T00:19:47.8578400Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:19:47.8578896Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:19:47.8579352Z cuda train torch_multimodal_clip 2024-08-21T00:20:30.7123960Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_157) 2024-08-21T00:20:50.3850950Z W0821 00:20:50.384000 60497 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:21:58.5396639Z pass 2024-08-21T00:21:58.6322203Z TIMING: entire_frame_compile:104.05725 code_gen:48.2914 inductor_compile:73.86258 backend_compile:84.66166 total_wall_time:122.00916 2024-08-21T00:21:58.6323670Z STATS: call_* op count: 1404 | FakeTensorMode.__torch_dispatch__:88998 | FakeTensor.__torch_dispatch__:24629 | ProxyTorchDispatchMode.__torch_dispatch__:40037 2024-08-21T00:21:58.6324811Z Dynamo produced 3 graphs covering 1404 ops with 7 graph breaks (5 unique) 2024-08-21T00:22:07.9663456Z 2024-08-21T00:22:08.6476351Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:08.6476844Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:08.6477289Z cuda train tts_angular 2024-08-21T00:22:12.2849237Z W0821 00:22:12.284000 61097 torch/_logging/_internal.py:1058] [10/0] Profiler function will be ignored 2024-08-21T00:22:17.7751123Z pass 2024-08-21T00:22:17.7816766Z TIMING: entire_frame_compile:6.88741 inductor_compile:5.22598 backend_compile:6.05246 code_gen:3.89608 total_wall_time:7.16969 2024-08-21T00:22:17.7818171Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:2216 | ProxyTorchDispatchMode.__torch_dispatch__:871 | FakeTensor.__torch_dispatch__:906 2024-08-21T00:22:17.7819330Z Dynamo produced 5 graphs covering 70 ops with 9 graph breaks (6 unique) 2024-08-21T00:22:21.7870292Z 2024-08-21T00:22:24.4464915Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:24.4465390Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:22:24.4469201Z cuda train vgg16 2024-08-21T00:22:59.4326759Z pass 2024-08-21T00:22:59.4562447Z TIMING: entire_frame_compile:6.53457 code_gen:6.47853 inductor_compile:7.66944 backend_compile:5.57984 total_wall_time:12.21515 2024-08-21T00:22:59.4564538Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:4569 | ProxyTorchDispatchMode.__torch_dispatch__:1686 | FakeTensor.__torch_dispatch__:606 2024-08-21T00:22:59.4566181Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-08-21T00:23:03.6663523Z 2024-08-21T00:23:07.0702316Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:23:07.0704106Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:23:07.0704578Z cuda train vision_maskrcnn 2024-08-21T00:23:41.6560111Z W0821 00:23:41.655000 61652 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:23:41.6565967Z W0821 00:23:41.656000 61652 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:23:41.6572636Z W0821 00:23:41.656000 61652 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:23:41.6579495Z W0821 00:23:41.657000 61652 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:23:41.6586059Z W0821 00:23:41.658000 61652 torch/_inductor/utils.py:1391] [5/0_1] DeviceCopy in input program 2024-08-21T00:23:43.7065803Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 2024-08-21T00:23:43.7068245Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward 2024-08-21T00:23:43.7069797Z anchors = self.anchor_generator(images, features) 2024-08-21T00:23:43.7071986Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward 2024-08-21T00:23:43.7073513Z self.set_cell_anchors(dtype, device) 2024-08-21T00:23:43.7075266Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors 2024-08-21T00:23:43.7077299Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-08-21T00:23:43.7079468Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in 2024-08-21T00:23:43.7081376Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-08-21T00:23:43.7082356Z 2024-08-21T00:23:45.4647793Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-08-21T00:23:45.4649469Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-21T00:23:45.4650495Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] or: 2024-08-21T00:23:45.4651493Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-21T00:23:45.4652707Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] to include these operations in the captured graph. 2024-08-21T00:23:45.4653714Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-21T00:23:45.4654917Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break: from user code at: 2024-08-21T00:23:45.4657083Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104 2024-08-21T00:23:45.4659155Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets) 2024-08-21T00:23:45.4661149Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward 2024-08-21T00:23:45.4662864Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] box_features = self.box_roi_pool(features, proposals, image_shapes) 2024-08-21T00:23:45.4664839Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward 2024-08-21T00:23:45.4666376Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] self.scales, self.map_levels = _setup_scales( 2024-08-21T00:23:45.4668566Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales 2024-08-21T00:23:45.4670242Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-21T00:23:45.4672059Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in 2024-08-21T00:23:45.4673717Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-21T00:23:45.4675508Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-21T00:23:45.4677297Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-21T00:23:45.4678344Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-21T00:23:45.4679083Z W0821 00:23:45.463000 61652 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-21T00:24:00.4546067Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:24:00.4547541Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:24:00.4548700Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:24:00.4549302Z 2024-08-21T00:24:06.3201584Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:24:06.3202966Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:24:06.3204143Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:24:06.3204730Z 2024-08-21T00:24:12.1550570Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:24:12.1552042Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:24:12.1553188Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:24:12.1553777Z 2024-08-21T00:24:50.0482125Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:24:50.0483485Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:24:50.0484631Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:24:50.0485322Z 2024-08-21T00:24:59.6155042Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:24:59.6156933Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:24:59.6158591Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:24:59.6159480Z 2024-08-21T00:25:13.1835746Z W0821 00:25:13.182000 61652 torch/fx/experimental/symbolic_shapes.py:4543] [30/6] RecursionError in sympy.xreplace(Eq(Mod(2*s3, s4), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-08-21T00:25:15.3486732Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:15.3488115Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:25:15.3489261Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:25:15.3490200Z 2024-08-21T00:25:19.8101270Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:19.8102601Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:25:19.8103736Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:25:19.8104327Z 2024-08-21T00:25:26.3431330Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:26.3432980Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-08-21T00:25:26.3434572Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-08-21T00:25:26.3435066Z 2024-08-21T00:25:27.3288468Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:27.3290050Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-08-21T00:25:27.3291638Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-08-21T00:25:27.3292127Z 2024-08-21T00:25:28.4620003Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:28.4621614Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804 2024-08-21T00:25:28.4622738Z masks_probs = maskrcnn_inference(mask_logits, labels) 2024-08-21T00:25:28.4623888Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference 2024-08-21T00:25:28.4624885Z mask_prob = mask_prob[index, labels][:, None] 2024-08-21T00:25:28.4625219Z 2024-08-21T00:25:31.6819167Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:31.6820941Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:25:31.6822103Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:25:31.6822697Z 2024-08-21T00:25:35.4204357Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:35.4205936Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward 2024-08-21T00:25:35.4206923Z features = self.backbone(images.tensors) 2024-08-21T00:25:35.4207985Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward 2024-08-21T00:25:35.4208872Z x = self.fpn(x) 2024-08-21T00:25:35.4209780Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward 2024-08-21T00:25:35.4210868Z inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest") 2024-08-21T00:25:35.4211340Z 2024-08-21T00:25:40.5977879Z W0821 00:25:40.597000 61652 torch/_dynamo/convert_frame.py:831] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-08-21T00:25:40.5979822Z W0821 00:25:40.597000 61652 torch/_dynamo/convert_frame.py:831] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-08-21T00:25:40.5981612Z W0821 00:25:40.597000 61652 torch/_dynamo/convert_frame.py:831] [30/8] last reason: 30/0: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 623 2024-08-21T00:25:40.5983037Z W0821 00:25:40.597000 61652 torch/_dynamo/convert_frame.py:831] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-08-21T00:25:40.5984582Z W0821 00:25:40.597000 61652 torch/_dynamo/convert_frame.py:831] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-08-21T00:25:48.4679406Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:48.4680807Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:25:48.4681932Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:25:48.4683191Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:25:48.4684076Z v1 = masked_index(y_low, x_low) 2024-08-21T00:25:48.4684999Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:25:48.4685813Z return input[ 2024-08-21T00:25:48.4686006Z 2024-08-21T00:25:53.8446884Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:25:53.8448356Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:25:53.8449848Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:25:53.8451117Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:25:53.8452004Z v1 = masked_index(y_low, x_low) 2024-08-21T00:25:53.8452920Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:25:53.8453728Z return input[ 2024-08-21T00:25:53.8453918Z 2024-08-21T00:26:00.5058033Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:26:00.5059521Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:26:00.5060636Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:26:00.5061888Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:26:00.5062806Z v1 = masked_index(y_low, x_low) 2024-08-21T00:26:00.5063722Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:26:00.5064529Z return input[ 2024-08-21T00:26:00.5064718Z 2024-08-21T00:26:12.9082476Z W0821 00:26:12.907000 61652 torch/fx/experimental/symbolic_shapes.py:4543] [54/4] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-08-21T00:26:14.4501781Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:26:14.4503226Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:26:14.4504369Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:26:14.4505685Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:26:14.4506595Z v1 = masked_index(y_low, x_low) 2024-08-21T00:26:14.4507687Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:26:14.4508513Z return input[ 2024-08-21T00:26:14.4508715Z 2024-08-21T00:26:25.2442015Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:26:25.2443513Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:26:25.2446751Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:26:25.2448031Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:26:25.2448924Z v1 = masked_index(y_low, x_low) 2024-08-21T00:26:25.2450212Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:26:25.2451038Z return input[ 2024-08-21T00:26:25.2451237Z 2024-08-21T00:26:34.0797204Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:26:34.0798728Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:26:34.0799851Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:26:34.0801108Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:26:34.0801999Z v1 = masked_index(y_low, x_low) 2024-08-21T00:26:34.0802915Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:26:34.0803721Z return input[ 2024-08-21T00:26:34.0803941Z 2024-08-21T00:26:38.2852852Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:26:38.2856153Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:26:38.2858372Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:26:38.2860858Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:26:38.2861797Z v1 = masked_index(y_low, x_low) 2024-08-21T00:26:38.2862705Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:26:38.2863522Z return input[ 2024-08-21T00:26:38.2863720Z 2024-08-21T00:26:39.4221035Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:26:39.4222945Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:26:39.4224097Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:26:39.4225372Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:26:39.4226265Z v1 = masked_index(y_low, x_low) 2024-08-21T00:26:39.4227286Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:26:39.4228105Z return input[ 2024-08-21T00:26:39.4228296Z 2024-08-21T00:26:39.7658095Z pass 2024-08-21T00:26:39.8282072Z TIMING: entire_frame_compile:192.99257 inductor_compile:93.34951 backend_compile:146.28811 code_gen:56.29654 total_wall_time:227.55846 2024-08-21T00:26:39.8284046Z STATS: call_* op count: 2436 | FakeTensorMode.__torch_dispatch__:137862 | FakeTensor.__torch_dispatch__:12951 | ProxyTorchDispatchMode.__torch_dispatch__:45360 | attempt fast:2589 | slow no contiguity match:274 | fast is_contiguous:1813 | slow both tensors nontrivially broadcast:502 2024-08-21T00:26:39.8285745Z Dynamo produced 51 graphs covering 2436 ops with 33 graph breaks (8 unique) 2024-08-21T00:26:50.6863233Z 2024-08-21T00:26:53.6071880Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:26:53.6072481Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:26:53.6072959Z cuda train yolov3 2024-08-21T00:27:38.4051032Z W0821 00:27:38.404000 63132 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:27:38.4054341Z W0821 00:27:38.404000 63132 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:27:38.6911060Z W0821 00:27:38.690000 63132 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:27:38.6915693Z W0821 00:27:38.691000 63132 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:27:39.0021728Z W0821 00:27:39.001000 63132 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:27:39.0023883Z W0821 00:27:39.002000 63132 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:27:56.9613696Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_358). Found from : 2024-08-21T00:27:56.9614998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:27:56.9615991Z pred = mod(*cloned_inputs) 2024-08-21T00:27:56.9617105Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-08-21T00:27:56.9618188Z return self.forward_once(x) 2024-08-21T00:27:56.9619088Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once 2024-08-21T00:27:56.9619978Z yolo_out.append(module(x, out)) 2024-08-21T00:27:56.9620833Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-08-21T00:27:56.9621737Z self.create_grids((nx, ny), p.device) 2024-08-21T00:27:56.9623614Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-08-21T00:27:56.9624551Z self.anchor_vec = self.anchor_vec.to(device) 2024-08-21T00:27:56.9624891Z 2024-08-21T00:28:31.3115991Z W0821 00:28:31.310000 63132 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:30:33.3928841Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:33.3930205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:30:33.3931185Z pred = mod(*cloned_inputs) 2024-08-21T00:30:33.3932020Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-08-21T00:30:33.3934022Z return self.forward_once(x) 2024-08-21T00:30:33.3934921Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 292, in forward_once 2024-08-21T00:30:33.3935798Z x = module(x) 2024-08-21T00:30:33.3935989Z 2024-08-21T00:30:33.6766554Z pass 2024-08-21T00:30:33.7172129Z TIMING: entire_frame_compile:141.56426 code_gen:91.96727 inductor_compile:129.93058 backend_compile:110.66557 total_wall_time:198.68189 2024-08-21T00:30:33.7173675Z STATS: call_* op count: 1377 | FakeTensorMode.__torch_dispatch__:106674 | FakeTensor.__torch_dispatch__:34279 | ProxyTorchDispatchMode.__torch_dispatch__:44439 2024-08-21T00:30:33.7174821Z Dynamo produced 4 graphs covering 1377 ops with 8 graph breaks (5 unique) 2024-08-21T00:30:43.8926795Z accuracy pass_rate=78.95% 2024-08-21T00:30:43.8930338Z calls_captured gmean=0.00x mean=431.184x 2024-08-21T00:30:43.8934529Z unique_graphs gmean=0.00x mean=3.816x 2024-08-21T00:30:43.8937900Z graph_breaks gmean=0.00x mean=6.395x 2024-08-21T00:30:43.8942422Z unique_graph_breaks gmean=0.00x mean=4.316x 2024-08-21T00:30:43.8946402Z autograd_captures gmean=0.00x mean=0.000x 2024-08-21T00:30:43.8950404Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-21T00:30:43.8953971Z cudagraph_skips gmean=0.00x mean=0.842x 2024-08-21T00:30:45.7113686Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv 2024-08-21T00:30:46.0488605Z llava XFAIL 2024-08-21T00:30:46.0492463Z maml_omniglot PASS 2024-08-21T00:30:46.0497012Z mnasnet1_0 PASS 2024-08-21T00:30:46.0501189Z mobilenet_v2 PASS 2024-08-21T00:30:46.0505563Z mobilenet_v2_quantized_qat XFAIL 2024-08-21T00:30:46.0510419Z mobilenet_v3_large PASS 2024-08-21T00:30:46.0514696Z moco PASS 2024-08-21T00:30:46.0519150Z nanogpt PASS 2024-08-21T00:30:46.0523653Z nvidia_deeprecommender PASS 2024-08-21T00:30:46.0527831Z opacus_cifar10 XFAIL 2024-08-21T00:30:46.0532183Z phlippe_densenet PASS 2024-08-21T00:30:46.0537161Z phlippe_resnet PASS 2024-08-21T00:30:46.0541609Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-21T00:30:46.0545901Z pytorch_stargan PASS 2024-08-21T00:30:46.0550499Z pytorch_unet XFAIL 2024-08-21T00:30:46.0554728Z resnet152 PASS 2024-08-21T00:30:46.0559073Z resnet18 PASS 2024-08-21T00:30:46.0563435Z resnet50 PASS 2024-08-21T00:30:46.0567785Z resnet50_quantized_qat XFAIL 2024-08-21T00:30:46.0572087Z resnext50_32x4d PASS 2024-08-21T00:30:46.0576447Z sam XFAIL 2024-08-21T00:30:46.0580797Z shufflenet_v2_x1_0 PASS 2024-08-21T00:30:46.0585455Z soft_actor_critic PASS 2024-08-21T00:30:46.0589977Z speech_transformer PASS 2024-08-21T00:30:46.0594066Z squeezenet1_1 PASS 2024-08-21T00:30:46.0598401Z stable_diffusion_text_encoder PASS 2024-08-21T00:30:46.0602722Z stable_diffusion_unet XFAIL 2024-08-21T00:30:46.0607046Z timm_efficientnet PASS 2024-08-21T00:30:46.0611473Z timm_regnet PASS 2024-08-21T00:30:46.0615717Z timm_resnest PASS 2024-08-21T00:30:46.0620060Z timm_vision_transformer PASS 2024-08-21T00:30:46.0624359Z timm_vision_transformer_large XFAIL 2024-08-21T00:30:46.0628903Z timm_vovnet PASS 2024-08-21T00:30:46.0633104Z torch_multimodal_clip PASS 2024-08-21T00:30:46.0638094Z tts_angular PASS 2024-08-21T00:30:46.0642316Z vgg16 PASS 2024-08-21T00:30:46.0646656Z vision_maskrcnn PASS 2024-08-21T00:30:46.0651063Z yolov3 PASS 2024-08-21T00:30:46.1176496Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv 2024-08-21T00:30:46.4551540Z llava PASS 2024-08-21T00:30:46.4556769Z maml_omniglot PASS 2024-08-21T00:30:46.4561040Z mnasnet1_0 PASS 2024-08-21T00:30:46.4565752Z mobilenet_v2 PASS 2024-08-21T00:30:46.4570870Z mobilenet_v2_quantized_qat PASS 2024-08-21T00:30:46.4575730Z mobilenet_v3_large PASS 2024-08-21T00:30:46.4580467Z moco PASS 2024-08-21T00:30:46.4585423Z nanogpt PASS 2024-08-21T00:30:46.4590786Z nvidia_deeprecommender PASS 2024-08-21T00:30:46.4595580Z opacus_cifar10 PASS 2024-08-21T00:30:46.4600303Z phlippe_densenet PASS 2024-08-21T00:30:46.4604984Z phlippe_resnet PASS 2024-08-21T00:30:46.4609804Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-21T00:30:46.4614619Z pytorch_stargan PASS 2024-08-21T00:30:46.4619311Z pytorch_unet PASS 2024-08-21T00:30:46.4624156Z resnet152 PASS 2024-08-21T00:30:46.4629186Z resnet18 PASS 2024-08-21T00:30:46.4634249Z resnet50 PASS 2024-08-21T00:30:46.4639186Z resnet50_quantized_qat PASS 2024-08-21T00:30:46.4643895Z resnext50_32x4d PASS 2024-08-21T00:30:46.4648789Z sam PASS 2024-08-21T00:30:46.4653742Z shufflenet_v2_x1_0 PASS 2024-08-21T00:30:46.4658429Z soft_actor_critic PASS 2024-08-21T00:30:46.4663288Z speech_transformer PASS 2024-08-21T00:30:46.4668267Z squeezenet1_1 PASS 2024-08-21T00:30:46.4673050Z stable_diffusion_text_encoder PASS 2024-08-21T00:30:46.4677774Z stable_diffusion_unet PASS 2024-08-21T00:30:46.4682575Z timm_efficientnet PASS 2024-08-21T00:30:46.4687366Z timm_regnet PASS 2024-08-21T00:30:46.4692142Z timm_resnest PASS 2024-08-21T00:30:46.4696917Z timm_vision_transformer PASS 2024-08-21T00:30:46.4701706Z timm_vision_transformer_large PASS 2024-08-21T00:30:46.4706748Z timm_vovnet PASS 2024-08-21T00:30:46.4711526Z torch_multimodal_clip PASS 2024-08-21T00:30:46.4716229Z tts_angular PASS 2024-08-21T00:30:46.4721025Z vgg16 PASS 2024-08-21T00:30:46.4725852Z vision_maskrcnn PASS 2024-08-21T00:30:46.4730668Z yolov3 PASS 2024-08-21T00:30:46.5258503Z + cleanup_workspace 2024-08-21T00:30:46.5259557Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-21T00:30:46.5261096Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-21T00:30:46.5262173Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-21T00:30:46.5262896Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-21T00:30:46.5263809Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-21T00:30:46.5264796Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-21T00:30:46.5265564Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-21T00:30:47.3072069Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-21T00:30:47.3072590Z cat test/**/*_toprint.log || true 2024-08-21T00:30:47.3085678Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:47.3086200Z env: 2024-08-21T00:30:47.3086467Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.3086921Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.3087662Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.3088308Z ##[endgroup] 2024-08-21T00:30:47.3182934Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-21T00:30:47.3224079Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:30:47.3224769Z kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:30:47.3235583Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:47.3236091Z env: 2024-08-21T00:30:47.3236358Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.3236807Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.3237542Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.3238212Z MONITOR_SCRIPT_PID: 102657 2024-08-21T00:30:47.3238564Z ##[endgroup] 2024-08-21T00:30:47.3450543Z Prepare all required actions 2024-08-21T00:30:47.3450990Z Getting action download info 2024-08-21T00:30:47.5510692Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-21T00:30:47.7145534Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-21T00:30:47.7146004Z with: 2024-08-21T00:30:47.7146590Z file-suffix: test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571 2024-08-21T00:30:47.7147397Z s3-bucket: gha-artifacts 2024-08-21T00:30:47.7147741Z env: 2024-08-21T00:30:47.7148019Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.7148485Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.7149231Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.7149891Z ##[endgroup] 2024-08-21T00:30:47.7176427Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-21T00:30:47.7177066Z # Remove any previous test jsons if they exist 2024-08-21T00:30:47.7177753Z rm -f test-jsons-*.zip 2024-08-21T00:30:47.7178274Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-21T00:30:47.7188448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:47.7188948Z env: 2024-08-21T00:30:47.7189215Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.7189666Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.7190397Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.7191336Z FILE_SUFFIX: test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571 2024-08-21T00:30:47.7192016Z ##[endgroup] 2024-08-21T00:30:47.7484842Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-21T00:30:47.7537451Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-21T00:30:47.7542936Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-21T00:30:47.7558209Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-21T00:30:47.7569256Z adding: test/slow_tests.json (deflated 82%) 2024-08-21T00:30:47.7607293Z ##[group]Run # Remove any previous test reports if they exist 2024-08-21T00:30:47.7608100Z # Remove any previous test reports if they exist 2024-08-21T00:30:47.7609132Z rm -f test-reports-*.zip 2024-08-21T00:30:47.7609727Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-21T00:30:47.7618763Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:47.7619269Z env: 2024-08-21T00:30:47.7619559Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.7620021Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.7620755Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.7621732Z FILE_SUFFIX: test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571 2024-08-21T00:30:47.7622477Z ##[endgroup] 2024-08-21T00:30:47.7884400Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-08-21T00:30:47.7886488Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-08-21T00:30:47.7887686Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-21T00:30:47.7888668Z adding: test/test-reports/training_torchbench.csv (deflated 66%) 2024-08-21T00:30:47.7896162Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-08-21T00:30:47.7897333Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-21T00:30:47.7931384Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-21T00:30:47.7932013Z # Remove any previous usage logs if they exist 2024-08-21T00:30:47.7932515Z rm -f logs-*.zip 2024-08-21T00:30:47.7933768Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-21T00:30:47.7934605Z # so check to see if the file exists first 2024-08-21T00:30:47.7935106Z if [ -f 'usage_log.txt' ]; then 2024-08-21T00:30:47.7935636Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-21T00:30:47.7936129Z fi 2024-08-21T00:30:47.7936498Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-21T00:30:47.7937072Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-21T00:30:47.7937570Z fi 2024-08-21T00:30:47.7946341Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:47.7946923Z env: 2024-08-21T00:30:47.7947197Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.7947652Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.7948387Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.7949333Z FILE_SUFFIX: test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571 2024-08-21T00:30:47.7950190Z ##[endgroup] 2024-08-21T00:30:47.8064777Z adding: usage_log.txt (deflated 92%) 2024-08-21T00:30:47.8154923Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-21T00:30:47.8155638Z # Remove any previous debugging artifacts if they exist 2024-08-21T00:30:47.8156177Z rm -f debug-*.zip 2024-08-21T00:30:47.8156553Z if [ -d 'test/debug' ]; then 2024-08-21T00:30:47.8157058Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-21T00:30:47.8157527Z fi 2024-08-21T00:30:47.8165579Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:47.8166079Z env: 2024-08-21T00:30:47.8166353Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.8166796Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.8167528Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.8168472Z FILE_SUFFIX: test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571 2024-08-21T00:30:47.8169162Z ##[endgroup] 2024-08-21T00:30:47.8264367Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:30:47.8264809Z with: 2024-08-21T00:30:47.8265087Z s3-bucket: gha-artifacts 2024-08-21T00:30:47.8265523Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:30:47.8265995Z retention-days: 14 2024-08-21T00:30:47.8266331Z if-no-files-found: warn 2024-08-21T00:30:47.8266765Z path: test-jsons-*.zip 2024-08-21T00:30:47.8267108Z name: artifact 2024-08-21T00:30:47.8267409Z region: us-east-1 2024-08-21T00:30:47.8267715Z env: 2024-08-21T00:30:47.8267982Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:47.8268434Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:47.8269171Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:47.8269814Z ##[endgroup] 2024-08-21T00:30:48.2190498Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:30:48.2191459Z With the provided path, there will be 1 file uploaded 2024-08-21T00:30:48.2192242Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:30:48.2253654Z Starting upload of test-jsons-test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571.zip 2024-08-21T00:30:48.4192141Z Finished upload of test-jsons-test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571.zip 2024-08-21T00:30:48.4484039Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:30:48.4484490Z with: 2024-08-21T00:30:48.4484773Z s3-bucket: gha-artifacts 2024-08-21T00:30:48.4485216Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:30:48.4485703Z retention-days: 14 2024-08-21T00:30:48.4486048Z if-no-files-found: error 2024-08-21T00:30:48.4486414Z path: test-reports-*.zip 2024-08-21T00:30:48.4486772Z name: artifact 2024-08-21T00:30:48.4487088Z region: us-east-1 2024-08-21T00:30:48.4487387Z env: 2024-08-21T00:30:48.4487880Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:48.4488359Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:48.4489108Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:48.4489755Z ##[endgroup] 2024-08-21T00:30:48.8071768Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:30:48.8072802Z With the provided path, there will be 1 file uploaded 2024-08-21T00:30:48.8073580Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:30:48.8127908Z Starting upload of test-reports-test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571.zip 2024-08-21T00:30:48.9008226Z Finished upload of test-reports-test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571.zip 2024-08-21T00:30:48.9297255Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:30:48.9297712Z with: 2024-08-21T00:30:48.9297997Z s3-bucket: gha-artifacts 2024-08-21T00:30:48.9298447Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:30:48.9299113Z retention-days: 14 2024-08-21T00:30:48.9299458Z if-no-files-found: ignore 2024-08-21T00:30:48.9299817Z path: logs-*.zip 2024-08-21T00:30:48.9300134Z name: artifact 2024-08-21T00:30:48.9300443Z region: us-east-1 2024-08-21T00:30:48.9300760Z env: 2024-08-21T00:30:48.9301036Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:48.9301504Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:48.9302260Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:48.9302923Z ##[endgroup] 2024-08-21T00:30:49.2920548Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:30:49.2921310Z With the provided path, there will be 1 file uploaded 2024-08-21T00:30:49.2921952Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:30:49.2978047Z Starting upload of logs-test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571.zip 2024-08-21T00:30:49.4405380Z Finished upload of logs-test-inductor_torchbench-2-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528571.zip 2024-08-21T00:30:49.4696440Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:30:49.4696889Z with: 2024-08-21T00:30:49.4697168Z s3-bucket: gha-artifacts 2024-08-21T00:30:49.4697606Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:30:49.4698084Z retention-days: 14 2024-08-21T00:30:49.4698425Z if-no-files-found: ignore 2024-08-21T00:30:49.4698783Z path: debug-*.zip 2024-08-21T00:30:49.4699097Z name: artifact 2024-08-21T00:30:49.4699403Z region: us-east-1 2024-08-21T00:30:49.4699694Z env: 2024-08-21T00:30:49.4699978Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:49.4700435Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:49.4701179Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:49.4701821Z ##[endgroup] 2024-08-21T00:30:49.8201365Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-21T00:30:49.8486331Z ##[group]Run # shellcheck disable=SC2156 2024-08-21T00:30:49.8486817Z # shellcheck disable=SC2156 2024-08-21T00:30:49.8487610Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-21T00:30:49.8496986Z shell: /usr/bin/bash -e {0} 2024-08-21T00:30:49.8497350Z env: 2024-08-21T00:30:49.8497623Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:49.8498081Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:49.8498819Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:49.8499468Z ##[endgroup] 2024-08-21T00:30:50.1228933Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-21T00:30:50.1229498Z with: 2024-08-21T00:30:50.1229759Z env: 2024-08-21T00:30:50.1230038Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:50.1230496Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:50.1231245Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:50.1231887Z ##[endgroup] 2024-08-21T00:30:50.1251850Z ##[group]Run set -eou pipefail 2024-08-21T00:30:50.1252255Z set -eou pipefail 2024-08-21T00:30:50.1252607Z  2024-08-21T00:30:50.1253115Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-21T00:30:50.1253749Z for _ in $(seq 1440); do 2024-08-21T00:30:50.1254228Z  # Break if no ssh session exists anymore 2024-08-21T00:30:50.1254731Z  if [ "$(who)" = "" ]; then 2024-08-21T00:30:50.1255139Z  break 2024-08-21T00:30:50.1255477Z  fi 2024-08-21T00:30:50.1255771Z  echo "." 2024-08-21T00:30:50.1256094Z  sleep 5 2024-08-21T00:30:50.1256407Z done 2024-08-21T00:30:50.1265129Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:50.1265615Z env: 2024-08-21T00:30:50.1266068Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:50.1266518Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:50.1267338Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:50.1267977Z ##[endgroup] 2024-08-21T00:30:50.1297958Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-21T00:30:50.1374782Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:30:50.1375554Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:30:50.1376138Z # shellcheck disable=SC2046 2024-08-21T00:30:50.1376612Z docker stop $(docker ps -q) || true 2024-08-21T00:30:50.1377091Z # Prune all of the docker images 2024-08-21T00:30:50.1377536Z docker system prune -af 2024-08-21T00:30:50.1388310Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:30:50.1388813Z env: 2024-08-21T00:30:50.1389096Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:30:50.1389556Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:30:50.1390296Z DOCKER_CONTAINER_ID: 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:50.1390942Z ##[endgroup] 2024-08-21T00:30:50.9918754Z 6d4c93a2f182 2024-08-21T00:30:55.0206733Z Deleted Containers: 2024-08-21T00:30:55.0207381Z 6d4c93a2f182d8b5439f7ee20683a1ee68d2f9b47e15aaad520f710f25b65803 2024-08-21T00:30:55.0207833Z 2024-08-21T00:31:04.0931576Z Deleted Images: 2024-08-21T00:31:04.0935203Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T00:31:04.0937558Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-21T00:31:04.0939377Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-21T00:31:04.0940431Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-21T00:31:04.0941464Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-21T00:31:04.0942478Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-21T00:31:04.0943462Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-21T00:31:04.0944658Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-21T00:31:04.0945663Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-21T00:31:04.0987484Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-21T00:31:04.0988667Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-21T00:31:04.0989726Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-21T00:31:04.0990828Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-21T00:31:04.0991873Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-21T00:31:04.0992970Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-21T00:31:04.0994017Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-21T00:31:04.0996595Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-21T00:31:04.0997656Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-21T00:31:04.0998717Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-21T00:31:04.0999744Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-21T00:31:04.1000795Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-21T00:31:04.1001852Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-21T00:31:04.1003187Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-21T00:31:04.1004250Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-21T00:31:04.1005298Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-21T00:31:04.1006369Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-21T00:31:04.1007420Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-21T00:31:04.1008467Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-21T00:31:04.1009524Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-21T00:31:04.1010606Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-21T00:31:04.1011676Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-21T00:31:04.1012789Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-21T00:31:04.1013829Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-21T00:31:04.1014869Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-21T00:31:04.1015920Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-21T00:31:04.1016952Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-21T00:31:04.1017990Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-21T00:31:04.1019054Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-21T00:31:04.1020122Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-21T00:31:04.1021166Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-21T00:31:04.1022209Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-21T00:31:04.1023263Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-21T00:31:04.1024314Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-21T00:31:04.1025370Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-21T00:31:04.1026396Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-21T00:31:04.1027668Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-21T00:31:04.1028888Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-21T00:31:04.1029955Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-21T00:31:04.1031014Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-21T00:31:04.1032051Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-21T00:31:04.1033082Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-21T00:31:04.1034378Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-21T00:31:04.1035429Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-21T00:31:04.1036447Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-21T00:31:04.1037477Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-21T00:31:04.1038615Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-21T00:31:04.1039656Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-21T00:31:04.1040691Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-21T00:31:04.1041721Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-21T00:31:04.1043005Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-21T00:31:04.1044059Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-21T00:31:04.1045129Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-21T00:31:04.1046207Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-21T00:31:04.1047281Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-21T00:31:04.1048404Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-21T00:31:04.1049454Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-21T00:31:04.1050511Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-21T00:31:04.1051596Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-21T00:31:04.1052669Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-21T00:31:04.1053758Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-21T00:31:04.1054816Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-21T00:31:04.1055881Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-21T00:31:04.1056934Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-21T00:31:04.1057987Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-21T00:31:04.1059035Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-21T00:31:04.1060116Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-21T00:31:04.1061196Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-21T00:31:04.1062273Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-21T00:31:04.1063380Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-21T00:31:04.1064449Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-21T00:31:04.1065512Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-21T00:31:04.1066142Z 2024-08-21T00:31:04.1066355Z Total reclaimed space: 49.95GB 2024-08-21T00:31:04.1145017Z Post job cleanup. 2024-08-21T00:31:04.1202508Z Post job cleanup. 2024-08-21T00:31:04.2145628Z [command]/usr/bin/git version 2024-08-21T00:31:04.2227510Z git version 2.40.1 2024-08-21T00:31:04.2265869Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/1d52ad7a-69e8-4a00-b896-d1a49ccc327e' before making global git config changes 2024-08-21T00:31:04.2267355Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T00:31:04.2270586Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-21T00:31:04.2319697Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T00:31:04.2366382Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T00:31:04.2770914Z Entering 'android/libs/fbjni' 2024-08-21T00:31:04.2840745Z Entering 'third_party/FP16' 2024-08-21T00:31:04.2909640Z Entering 'third_party/FXdiv' 2024-08-21T00:31:04.2978345Z Entering 'third_party/NNPACK' 2024-08-21T00:31:04.3048005Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:31:04.3116320Z Entering 'third_party/XNNPACK' 2024-08-21T00:31:04.3201595Z Entering 'third_party/benchmark' 2024-08-21T00:31:04.3270852Z Entering 'third_party/cpp-httplib' 2024-08-21T00:31:04.3339590Z Entering 'third_party/cpuinfo' 2024-08-21T00:31:04.3408865Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:31:04.3477486Z Entering 'third_party/cutlass' 2024-08-21T00:31:04.3552852Z Entering 'third_party/eigen' 2024-08-21T00:31:04.3622526Z Entering 'third_party/fbgemm' 2024-08-21T00:31:04.3691092Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:31:04.3759245Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:31:04.3825138Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:31:04.3902796Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:31:04.3977955Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:31:04.4048402Z Entering 'third_party/flatbuffers' 2024-08-21T00:31:04.4120495Z Entering 'third_party/fmt' 2024-08-21T00:31:04.4189836Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:31:04.4259205Z Entering 'third_party/gloo' 2024-08-21T00:31:04.4326778Z Entering 'third_party/googletest' 2024-08-21T00:31:04.4396261Z Entering 'third_party/ideep' 2024-08-21T00:31:04.4461711Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:31:04.4537635Z Entering 'third_party/ittapi' 2024-08-21T00:31:04.4606827Z Entering 'third_party/kineto' 2024-08-21T00:31:04.4677775Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:31:04.4743016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:31:04.4812376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:31:04.4879578Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:31:04.4947621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:31:04.5012138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:31:04.5084831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:31:04.5152320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:31:04.5219259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:31:04.5288389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:31:04.5359469Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:31:04.5426142Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:31:04.5498802Z Entering 'third_party/mimalloc' 2024-08-21T00:31:04.5569091Z Entering 'third_party/nccl/nccl' 2024-08-21T00:31:04.5637100Z Entering 'third_party/nlohmann' 2024-08-21T00:31:04.5705928Z Entering 'third_party/onnx' 2024-08-21T00:31:04.5786788Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:31:04.5854620Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:31:04.5932787Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:31:04.5999624Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:31:04.6066887Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:31:04.6136235Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:31:04.6202742Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:31:04.6271021Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:31:04.6337137Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:31:04.6407532Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:31:04.6472498Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:31:04.6543518Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:31:04.6613878Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:31:04.6704783Z Entering 'third_party/pocketfft' 2024-08-21T00:31:04.6773670Z Entering 'third_party/protobuf' 2024-08-21T00:31:04.6842716Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:31:04.6908568Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:31:04.6978628Z Entering 'third_party/psimd' 2024-08-21T00:31:04.7048651Z Entering 'third_party/pthreadpool' 2024-08-21T00:31:04.7117746Z Entering 'third_party/pybind11' 2024-08-21T00:31:04.7187286Z Entering 'third_party/python-peachpy' 2024-08-21T00:31:04.7254569Z Entering 'third_party/sleef' 2024-08-21T00:31:04.7325749Z Entering 'third_party/tensorpipe' 2024-08-21T00:31:04.7393023Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:31:04.7458987Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:31:04.7525890Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:31:04.7595161Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:31:04.7659857Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:31:04.7755929Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T00:31:04.7791873Z http.https://github.com/.extraheader 2024-08-21T00:31:04.7801660Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-21T00:31:04.7848902Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T00:31:04.8213227Z Entering 'android/libs/fbjni' 2024-08-21T00:31:04.8261259Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8304994Z Entering 'third_party/FP16' 2024-08-21T00:31:04.8348949Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8392586Z Entering 'third_party/FXdiv' 2024-08-21T00:31:04.8438385Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8484241Z Entering 'third_party/NNPACK' 2024-08-21T00:31:04.8528548Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8576400Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:31:04.8621125Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8666329Z Entering 'third_party/XNNPACK' 2024-08-21T00:31:04.8708898Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8771457Z Entering 'third_party/benchmark' 2024-08-21T00:31:04.8815333Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8859018Z Entering 'third_party/cpp-httplib' 2024-08-21T00:31:04.8901621Z http.https://github.com/.extraheader 2024-08-21T00:31:04.8945018Z Entering 'third_party/cpuinfo' 2024-08-21T00:31:04.8987921Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9031399Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:31:04.9075726Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9119745Z Entering 'third_party/cutlass' 2024-08-21T00:31:04.9164329Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9215794Z Entering 'third_party/eigen' 2024-08-21T00:31:04.9260016Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9306182Z Entering 'third_party/fbgemm' 2024-08-21T00:31:04.9350264Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9393381Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:31:04.9440640Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9483566Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:31:04.9526074Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9570499Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:31:04.9614551Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9664857Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:31:04.9713408Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9758263Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:31:04.9800451Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9846707Z Entering 'third_party/flatbuffers' 2024-08-21T00:31:04.9890742Z http.https://github.com/.extraheader 2024-08-21T00:31:04.9939227Z Entering 'third_party/fmt' 2024-08-21T00:31:04.9982475Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0026087Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:31:05.0072190Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0116212Z Entering 'third_party/gloo' 2024-08-21T00:31:05.0160023Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0203906Z Entering 'third_party/googletest' 2024-08-21T00:31:05.0247304Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0291843Z Entering 'third_party/ideep' 2024-08-21T00:31:05.0336562Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0379430Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:31:05.0421145Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0473498Z Entering 'third_party/ittapi' 2024-08-21T00:31:05.0517275Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0561804Z Entering 'third_party/kineto' 2024-08-21T00:31:05.0605123Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0648938Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:31:05.0691434Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0737061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:31:05.0783330Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0832307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:31:05.0876203Z http.https://github.com/.extraheader 2024-08-21T00:31:05.0921046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:31:05.0964780Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1009912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:31:05.1053827Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1097703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:31:05.1141354Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1188840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:31:05.1231581Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1277481Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:31:05.1320159Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1367001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:31:05.1409669Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1456046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:31:05.1499585Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1547247Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:31:05.1589932Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1633046Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:31:05.1676390Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1723597Z Entering 'third_party/mimalloc' 2024-08-21T00:31:05.1769083Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1813246Z Entering 'third_party/nccl/nccl' 2024-08-21T00:31:05.1858439Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1903359Z Entering 'third_party/nlohmann' 2024-08-21T00:31:05.1948198Z http.https://github.com/.extraheader 2024-08-21T00:31:05.1993485Z Entering 'third_party/onnx' 2024-08-21T00:31:05.2037880Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2099111Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:31:05.2143044Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2187047Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:31:05.2230116Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2278546Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:31:05.2322247Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2371059Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:31:05.2412744Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2456725Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:31:05.2499403Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2543629Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:31:05.2590717Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2632812Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:31:05.2675400Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2720186Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:31:05.2762924Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2805134Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:31:05.2848461Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2893592Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:31:05.2937503Z http.https://github.com/.extraheader 2024-08-21T00:31:05.2979679Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:31:05.3022346Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3069986Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:31:05.3111884Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3158743Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:31:05.3200905Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3265190Z Entering 'third_party/pocketfft' 2024-08-21T00:31:05.3309575Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3357776Z Entering 'third_party/protobuf' 2024-08-21T00:31:05.3402394Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3449309Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:31:05.3491550Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3536633Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:31:05.3581093Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3628781Z Entering 'third_party/psimd' 2024-08-21T00:31:05.3673190Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3718478Z Entering 'third_party/pthreadpool' 2024-08-21T00:31:05.3762929Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3807036Z Entering 'third_party/pybind11' 2024-08-21T00:31:05.3850967Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3895126Z Entering 'third_party/python-peachpy' 2024-08-21T00:31:05.3939473Z http.https://github.com/.extraheader 2024-08-21T00:31:05.3985694Z Entering 'third_party/sleef' 2024-08-21T00:31:05.4033314Z http.https://github.com/.extraheader 2024-08-21T00:31:05.4079121Z Entering 'third_party/tensorpipe' 2024-08-21T00:31:05.4123379Z http.https://github.com/.extraheader 2024-08-21T00:31:05.4166877Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:31:05.4209130Z http.https://github.com/.extraheader 2024-08-21T00:31:05.4251819Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:31:05.4295533Z http.https://github.com/.extraheader 2024-08-21T00:31:05.4342435Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:31:05.4385108Z http.https://github.com/.extraheader 2024-08-21T00:31:05.4428821Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:31:05.4471230Z http.https://github.com/.extraheader 2024-08-21T00:31:05.4513527Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:31:05.4557257Z http.https://github.com/.extraheader 2024-08-21T00:31:05.4718172Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-21T00:31:05.4740089Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-21T00:31:05.4748265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:31:05.4748778Z ##[endgroup] 2024-08-21T00:31:12.4417453Z Cleaning up orphan processes