2024-08-20T22:09:57.2194790Z Current runner version: '2.319.1' 2024-08-20T22:09:57.2201551Z Runner name: 'i-0d6323d38c97c2036' 2024-08-20T22:09:57.2202360Z Runner group name: 'Default' 2024-08-20T22:09:57.2203292Z Machine name: 'ip-10-0-43-152' 2024-08-20T22:09:57.2220030Z Testing runner upgrade compatibility 2024-08-20T22:09:57.4237110Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:09:57.4239112Z Actions: read 2024-08-20T22:09:57.4239610Z Attestations: read 2024-08-20T22:09:57.4240226Z Checks: read 2024-08-20T22:09:57.4240734Z Contents: read 2024-08-20T22:09:57.4241304Z Deployments: read 2024-08-20T22:09:57.4241791Z Discussions: read 2024-08-20T22:09:57.4242328Z Issues: read 2024-08-20T22:09:57.4242872Z Metadata: read 2024-08-20T22:09:57.4243359Z Packages: read 2024-08-20T22:09:57.4243862Z Pages: read 2024-08-20T22:09:57.4244464Z PullRequests: read 2024-08-20T22:09:57.4245095Z RepositoryProjects: read 2024-08-20T22:09:57.4245666Z SecurityEvents: read 2024-08-20T22:09:57.4246262Z Statuses: read 2024-08-20T22:09:57.4246960Z ##[endgroup] 2024-08-20T22:09:57.4250149Z Secret source: Actions 2024-08-20T22:09:57.4250856Z Prepare workflow directory 2024-08-20T22:09:57.5164647Z Prepare all required actions 2024-08-20T22:09:57.5328102Z Getting action download info 2024-08-20T22:09:57.6735925Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:09:58.1243034Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:10:02.1913615Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:10:02.3871220Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:10:02.6958592Z Getting action download info 2024-08-20T22:10:02.7890135Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:10:02.9684302Z Getting action download info 2024-08-20T22:10:03.0729153Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:10:03.2229672Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:10:03.2231945Z ##[group] Inputs 2024-08-20T22:10:03.2232418Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:10:03.2241603Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:10:03.2253135Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:03.2254292Z sync-tag: 2024-08-20T22:10:03.2255129Z timeout-minutes: 240 2024-08-20T22:10:03.2255475Z use-gha: 2024-08-20T22:10:03.2255773Z dashboard-tag: 2024-08-20T22:10:03.2256118Z s3-bucket: gha-artifacts 2024-08-20T22:10:03.2256514Z aws-role-to-assume: 2024-08-20T22:10:03.2256849Z ##[endgroup] 2024-08-20T22:10:03.2257751Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:10:03.2972542Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:03.3114634Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:10:03.3126172Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:03.3126697Z ##[endgroup] 2024-08-20T22:10:04.4909241Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:10:04.4909791Z Instance Type: g5.4xlarge 2024-08-20T22:10:04.4910472Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:10:04.4911032Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:10.2804861Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:10.2805426Z with: 2024-08-20T22:10:10.2806174Z github-secret: *** 2024-08-20T22:10:10.2807113Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:10.2808143Z activate-with-label: false 2024-08-20T22:10:10.2808508Z label: with-ssh 2024-08-20T22:10:10.2808827Z remove-existing-keys: true 2024-08-20T22:10:10.2809187Z fail-silently: true 2024-08-20T22:10:10.2809500Z env: 2024-08-20T22:10:10.2809777Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.2810117Z ##[endgroup] 2024-08-20T22:10:10.3691102Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:10.3694177Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:10.6345299Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:10.6992130Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:10.6992753Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:10.7878178Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:10.7893284Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:10.7916764Z Login using: ssh ec2-user@ec2-54-211-129-197.compute-1.amazonaws.com 2024-08-20T22:10:10.7917607Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:10.7918429Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:10.8051121Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:10.8051689Z with: 2024-08-20T22:10:10.8051971Z submodules: recursive 2024-08-20T22:10:10.8052414Z fetch-depth: 0 2024-08-20T22:10:10.8052710Z env: 2024-08-20T22:10:10.8052986Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.8053323Z ##[endgroup] 2024-08-20T22:10:10.8251262Z ##[group]Run retry () { 2024-08-20T22:10:10.8251657Z retry () { 2024-08-20T22:10:10.8252460Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:10.8253013Z } 2024-08-20T22:10:10.8253320Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.8253740Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:10.8254219Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.8254674Z else 2024-08-20T22:10:10.8255017Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.8255447Z fi 2024-08-20T22:10:10.8255792Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:10.8268681Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:10.8269197Z env: 2024-08-20T22:10:10.8269482Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.8269827Z NO_SUDO: 2024-08-20T22:10:10.8270110Z ##[endgroup] 2024-08-20T22:10:10.8306921Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:10.9928880Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:10.9929339Z with: 2024-08-20T22:10:10.9929665Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:10.9930108Z fetch-depth: 0 2024-08-20T22:10:10.9930422Z submodules: recursive 2024-08-20T22:10:10.9930756Z quiet-checkout: true 2024-08-20T22:10:10.9931103Z repository: pytorch/pytorch 2024-08-20T22:10:10.9931583Z token: *** 2024-08-20T22:10:10.9931874Z ssh-strict: true 2024-08-20T22:10:10.9932284Z persist-credentials: true 2024-08-20T22:10:10.9932646Z clean: true 2024-08-20T22:10:10.9932983Z sparse-checkout-cone-mode: true 2024-08-20T22:10:10.9933373Z lfs: false 2024-08-20T22:10:10.9933671Z set-safe-directory: true 2024-08-20T22:10:10.9934009Z env: 2024-08-20T22:10:10.9934282Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.9934612Z ##[endgroup] 2024-08-20T22:10:11.0895864Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:11.0897716Z ##[group]Getting Git version info 2024-08-20T22:10:11.0898674Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:11.0899713Z [command]/usr/bin/git version 2024-08-20T22:10:11.0900091Z git version 2.40.1 2024-08-20T22:10:11.0902139Z ##[endgroup] 2024-08-20T22:10:11.0916459Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c8691482-27a4-4376-b329-2015babf75e0' before making global git config changes 2024-08-20T22:10:11.0917739Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:11.0921622Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:11.0971778Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:11.0975940Z ##[group]Initializing the repository 2024-08-20T22:10:11.0978630Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:11.1023064Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:11.1024055Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:11.1024866Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:11.1025435Z hint: 2024-08-20T22:10:11.1026086Z hint: git config --global init.defaultBranch 2024-08-20T22:10:11.1026578Z hint: 2024-08-20T22:10:11.1027129Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:11.1028123Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:11.1029161Z hint: 2024-08-20T22:10:11.1029534Z hint: git branch -m 2024-08-20T22:10:11.1030295Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:11.1036609Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:11.1081820Z ##[endgroup] 2024-08-20T22:10:11.1083212Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:11.1084486Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:11.1124599Z ##[endgroup] 2024-08-20T22:10:11.1125128Z ##[group]Setting up auth 2024-08-20T22:10:11.1130616Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:11.1171526Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:11.1537015Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:11.1577397Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:11.1928219Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:11.1982126Z ##[endgroup] 2024-08-20T22:10:11.1982706Z ##[group]Fetching the repository 2024-08-20T22:10:11.1987893Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:14.1369661Z remote: Enumerating objects: 1008509 2024-08-20T22:10:14.1370358Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:14.1371036Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:14.1373989Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:14.1374736Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:14.1375287Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:14.1375961Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:14.1376606Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:14.1377241Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:14.1377904Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:14.1378616Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:14.1379391Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:14.1380177Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:14.1380931Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:14.1381692Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:14.1382204Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:14.1382722Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:14.1383441Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:14.1384227Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:14.1384975Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:14.1385527Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:14.1386152Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:14.1386906Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:14.1387479Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:14.1388010Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:14.1388501Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:14.1389038Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:14.1389552Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:14.1390050Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:14.1390561Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:14.1391574Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:14.1392355Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:14.1393043Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:14.1393720Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:14.1394386Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:14.1395017Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:14.1395728Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:14.1396224Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:14.1396720Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:14.1397217Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:14.1398056Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:14.1398544Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:14.1399053Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:14.1399563Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:14.1400062Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:14.1400556Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:14.1401056Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:14.1401576Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:14.1402074Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:14.1402568Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:14.1403065Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:14.1403561Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:14.1404060Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:14.1404549Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:14.1405047Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:14.1405542Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:14.1406040Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:14.1406527Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:14.1407025Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:14.1407522Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:14.1408018Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:14.1408519Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:14.1409051Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:14.1409575Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:14.1410096Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:14.1410625Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:14.1411136Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:14.1411655Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:14.1412293Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:14.1412809Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:14.1413321Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:14.1413838Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:14.1414353Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:14.1414866Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:14.1415376Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:14.1415895Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:14.1416406Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:14.1416915Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:14.1417420Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:14.1417934Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:14.1418446Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:14.1419150Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:14.1419661Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:14.1420172Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:14.1420686Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:14.1421195Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:14.1421701Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:14.1422210Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:14.1422838Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:14.1423349Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:14.1423853Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:14.1424361Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:14.1424872Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:14.1425384Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:14.1425897Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:14.1426408Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:14.1426917Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:14.1427424Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:14.1427936Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:14.1428446Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:14.1428969Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:14.1429473Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:14.1429986Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:14.1430536Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:14.1544543Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:14.1810176Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:14.2096647Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:14.2555107Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:14.3359428Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:14.3834894Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:14.4451983Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:14.4777777Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:14.5173720Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:14.5400800Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:14.5642265Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:14.5888980Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:14.6066464Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:14.6232213Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:14.6350366Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:14.6437332Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:14.6524553Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:14.6582278Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:14.6623595Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:14.6672192Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:14.6703413Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:14.6724087Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:14.6727960Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:14.6732017Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:14.6741248Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:14.6744901Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:14.6754774Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:14.6759461Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:14.6779949Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:14.6792209Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:14.6802967Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:14.6811702Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:14.6816302Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:14.6826714Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:14.6834989Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:14.6838733Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:14.6846951Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:14.6853160Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:14.6862611Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:14.6868945Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:14.6874678Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:14.6880059Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:14.6889085Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:14.6893040Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:14.6901491Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:14.6906190Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:14.6911259Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:14.6915346Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:14.6918267Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:14.6922012Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:14.6924593Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:14.6928057Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:14.6930679Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:14.6932008Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:14.6933567Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:14.6935530Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:14.6936503Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:14.6937780Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:14.6939205Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:14.6939921Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:14.6940500Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:14.6941350Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:14.6942043Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:14.6942588Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:14.6944434Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:14.6951118Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:14.6956365Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:14.6960654Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:14.6964635Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:14.6968585Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:14.6973618Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:14.6976090Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:14.6980685Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:14.6983026Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:14.6985503Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:14.6988571Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:14.6990895Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:14.6993954Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:14.6996525Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:14.6999071Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:14.7000249Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:14.7002545Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:14.7004325Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:14.7006595Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:14.7008765Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:14.7010011Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:14.7012293Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:14.7013428Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:14.7015513Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:14.7017379Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:14.7019292Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:14.7020281Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:14.7021425Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:14.7021967Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:14.7022917Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:14.7024011Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:14.7025797Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:14.7026768Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:14.7027896Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:14.7029246Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:14.7030441Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:14.7031011Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:38.1480218Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:11:01.5093840Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:11:01.5134149Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:01.5140213Z ##[endgroup] 2024-08-20T22:11:01.5144404Z ##[group]Determining the checkout info 2024-08-20T22:11:01.5145685Z ##[endgroup] 2024-08-20T22:11:01.5146408Z ##[group]Checking out the ref 2024-08-20T22:11:01.5147207Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:03.2283582Z ##[endgroup] 2024-08-20T22:11:03.2284186Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:03.2288930Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:03.2362787Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:03.2401831Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:03.2441030Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:03.2477270Z ##[endgroup] 2024-08-20T22:11:03.2477799Z ##[group]Fetching submodules 2024-08-20T22:11:03.2481085Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:03.2858219Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:03.3224989Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:03.3226849Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:03.3230377Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:03.3233613Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:03.3237477Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:03.3239880Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:03.3243592Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:03.3247133Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:03.3250700Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:03.3254774Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:03.3258468Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:03.3262163Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:03.3265954Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:03.3269989Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:03.3273792Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:03.3279314Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:03.3286569Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:03.3290939Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:03.3295204Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:03.3300494Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:03.3304954Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:03.3309397Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:03.3313747Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:03.3318383Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:03.3322996Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:03.3328105Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:03.3332699Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:03.3337820Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:03.3342731Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:03.3347829Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:03.3352832Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:03.3358237Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:03.3363244Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:03.3371696Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:03.3407403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:03.6423340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:03.8296995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:04.0102842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:04.2485892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:06.1882665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:17.5970398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:18.0079767Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:18.4569149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:19.0725329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:20.3731619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:22.4634308Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:28.3129872Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:29.7240146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:31.5285487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:32.8466502Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:33.2983097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:33.6629350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:34.8630616Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:35.2451508Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:35.5167814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:37.1349460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:37.8529729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:38.5158121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:44.5741589Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:46.6383627Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:50.9686840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:51.2148919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:12:00.3096203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:12:00.4832272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:12:00.6987205Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:12:01.6878052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:12:02.0755004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:12:02.7699333Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:03.2013846Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:03.2162619Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:03.2283510Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:03.2588924Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:03.3012626Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:04.4775352Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:04.5061229Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:04.5587989Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:04.6714567Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:04.7115843Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:05.3367680Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:05.6238069Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:05.7169358Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:05.7194741Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:05.7198621Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:05.7202376Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:05.7206088Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:05.7209958Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:05.7243562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:06.7377895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:07.3917741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:09.5732674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:10.5801432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:10.9651672Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:11.0759016Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:11.5904809Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:11.6613458Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:11.6762007Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:11.8230781Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:11.8685739Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:11.9142343Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:11.9466514Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:12.0000767Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:12.0162327Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:12.0181884Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:12.0212745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:25.3464523Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:25.3683291Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:25.4665500Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:25.4687265Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:25.4690414Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:25.4694103Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:25.4725507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:26.0481157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:27.2830629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:28.4521968Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:28.4541104Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:28.4544600Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:28.4548251Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:28.4551906Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:28.4555619Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:28.4559469Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:28.4563289Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:28.4567075Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:28.4599589Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:29.3448905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:29.7164599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:30.9634025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:31.3240632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:31.8774686Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:32.9187520Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:40.0265966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:40.4424248Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:40.4646471Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:40.5067692Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:40.5238576Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:40.5256771Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:40.5288621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:40.8368258Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:40.8588990Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:40.9072542Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:41.0254277Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:41.0450538Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:41.0893473Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:41.1561562Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:41.2008163Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:41.2307785Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:41.3527838Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:41.8625280Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:41.8662875Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:41.8666013Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:41.8697767Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:42.3345594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:43.3183055Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:43.3587722Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:43.4456623Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:43.4479640Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:43.4482847Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:43.4486033Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:43.4489586Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:43.4493570Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:43.4497128Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:43.4501431Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:43.4505282Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:43.4536751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:43.9342445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:45.0460482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:45.3499260Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:51.4200093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:51.6719051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:51.8741526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:52.1615985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:12:58.8802614Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:12:58.9279796Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:12:58.9466837Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:12:59.0698842Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:12:59.0858994Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:12:59.1044784Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:12:59.1244510Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:12:59.1263542Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:59.1267072Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:59.1298242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:13:01.0926909Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:13:02.4169405Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:13:02.4693412Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:13:03.1393191Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:13:03.1541824Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:13:03.4703405Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:13:03.4731215Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:03.4734701Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:03.4766327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:03.9795553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:05.0606147Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:05.1411262Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:05.1527018Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:05.1678362Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:05.2111229Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:05.2449207Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:05.2932012Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:05.3265887Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:05.3287851Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:05.3290851Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:05.3295146Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:05.3298480Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:05.3330931Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:06.3885165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:06.6101460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:07.8099432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:08.8045436Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:08.8240409Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:08.9013484Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:08.9359646Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:08.9378431Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:08.9410289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:09.1269387Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:09.1318689Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:09.1682086Z Entering 'android/libs/fbjni' 2024-08-20T22:13:09.1735039Z Entering 'third_party/FP16' 2024-08-20T22:13:09.1787817Z Entering 'third_party/FXdiv' 2024-08-20T22:13:09.1840954Z Entering 'third_party/NNPACK' 2024-08-20T22:13:09.1894191Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:09.1945147Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:09.2013406Z Entering 'third_party/benchmark' 2024-08-20T22:13:09.2064915Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:09.2116741Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:09.2170467Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:09.2223817Z Entering 'third_party/cutlass' 2024-08-20T22:13:09.2285341Z Entering 'third_party/eigen' 2024-08-20T22:13:09.2343274Z Entering 'third_party/fbgemm' 2024-08-20T22:13:09.2396659Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:09.2449173Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:09.2506516Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:09.2564182Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:09.2614795Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:09.2668100Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:09.2724489Z Entering 'third_party/fmt' 2024-08-20T22:13:09.2774232Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:09.2826233Z Entering 'third_party/gloo' 2024-08-20T22:13:09.2877632Z Entering 'third_party/googletest' 2024-08-20T22:13:09.2929698Z Entering 'third_party/ideep' 2024-08-20T22:13:09.2978951Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:09.3038295Z Entering 'third_party/ittapi' 2024-08-20T22:13:09.3090187Z Entering 'third_party/kineto' 2024-08-20T22:13:09.3142593Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:09.3195320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:09.3248815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:09.3300452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:09.3352115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:09.3400929Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:09.3460279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:09.3511731Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:09.3564057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:09.3614849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:09.3668866Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:09.3720398Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:09.3780431Z Entering 'third_party/mimalloc' 2024-08-20T22:13:09.3836280Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:09.3887274Z Entering 'third_party/nlohmann' 2024-08-20T22:13:09.3941540Z Entering 'third_party/onnx' 2024-08-20T22:13:09.4006879Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:09.4058149Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:09.4115943Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:09.4167719Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:09.4219551Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:09.4270931Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:09.4326231Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:09.4378505Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:09.4428636Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:09.4479071Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:09.4528536Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:09.4581338Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:09.4635257Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:09.4708318Z Entering 'third_party/pocketfft' 2024-08-20T22:13:09.4759383Z Entering 'third_party/protobuf' 2024-08-20T22:13:09.4816991Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:09.4867420Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:09.4922960Z Entering 'third_party/psimd' 2024-08-20T22:13:09.4974474Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:09.5025931Z Entering 'third_party/pybind11' 2024-08-20T22:13:09.5077664Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:09.5129608Z Entering 'third_party/sleef' 2024-08-20T22:13:09.5181497Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:09.5233768Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:09.5283642Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:09.5334209Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:09.5384782Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:09.5433957Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:09.5508776Z ##[endgroup] 2024-08-20T22:13:09.5511773Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:09.5514734Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:09.5876603Z Entering 'android/libs/fbjni' 2024-08-20T22:13:09.5942338Z Entering 'third_party/FP16' 2024-08-20T22:13:09.6014381Z Entering 'third_party/FXdiv' 2024-08-20T22:13:09.6081325Z Entering 'third_party/NNPACK' 2024-08-20T22:13:09.6148335Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:09.6215342Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:09.6302897Z Entering 'third_party/benchmark' 2024-08-20T22:13:09.6369835Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:09.6442007Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:09.6514330Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:09.6581337Z Entering 'third_party/cutlass' 2024-08-20T22:13:09.6656127Z Entering 'third_party/eigen' 2024-08-20T22:13:09.6727959Z Entering 'third_party/fbgemm' 2024-08-20T22:13:09.6794529Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:09.6860794Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:09.6927012Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:09.6999225Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:09.7065793Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:09.7134003Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:09.7207384Z Entering 'third_party/fmt' 2024-08-20T22:13:09.7274829Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:09.7341887Z Entering 'third_party/gloo' 2024-08-20T22:13:09.7408563Z Entering 'third_party/googletest' 2024-08-20T22:13:09.7477202Z Entering 'third_party/ideep' 2024-08-20T22:13:09.7544603Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:09.7620205Z Entering 'third_party/ittapi' 2024-08-20T22:13:09.7687275Z Entering 'third_party/kineto' 2024-08-20T22:13:09.7754142Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:09.7819626Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:09.7887349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:09.7956953Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:09.8024707Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:09.8089207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:09.8159691Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:09.8226810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:09.8294576Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:09.8363488Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:09.8439318Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:09.8506191Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:09.8576745Z Entering 'third_party/mimalloc' 2024-08-20T22:13:09.8644459Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:09.8711937Z Entering 'third_party/nlohmann' 2024-08-20T22:13:09.8779516Z Entering 'third_party/onnx' 2024-08-20T22:13:09.8861783Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:09.8930862Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:09.9004141Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:09.9071217Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:09.9138761Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:09.9205551Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:09.9271501Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:09.9338636Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:09.9406789Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:09.9472570Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:09.9537829Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:09.9606522Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:09.9678584Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:09.9766643Z Entering 'third_party/pocketfft' 2024-08-20T22:13:09.9838335Z Entering 'third_party/protobuf' 2024-08-20T22:13:09.9907548Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:09.9974824Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:10.0046418Z Entering 'third_party/psimd' 2024-08-20T22:13:10.0114502Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:10.0182054Z Entering 'third_party/pybind11' 2024-08-20T22:13:10.0249775Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:10.0317393Z Entering 'third_party/sleef' 2024-08-20T22:13:10.0384307Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:10.0450109Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:10.0516554Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:10.0582356Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:10.0650056Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:10.0714854Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:10.0808724Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:10.1172639Z Entering 'android/libs/fbjni' 2024-08-20T22:13:10.1234580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:10.1255755Z Entering 'third_party/FP16' 2024-08-20T22:13:10.1319527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:10.1340781Z Entering 'third_party/FXdiv' 2024-08-20T22:13:10.1404334Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:10.1424557Z Entering 'third_party/NNPACK' 2024-08-20T22:13:10.1487530Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:10.1509190Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:10.1570757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:10.1591958Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:10.1654465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:10.1691279Z Entering 'third_party/benchmark' 2024-08-20T22:13:10.1765448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:10.1775939Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:10.1836859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:10.1857300Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:10.1918305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:10.1939365Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:10.1999966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:10.2020856Z Entering 'third_party/cutlass' 2024-08-20T22:13:10.2081029Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:10.2109383Z Entering 'third_party/eigen' 2024-08-20T22:13:10.2169554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:10.2192556Z Entering 'third_party/fbgemm' 2024-08-20T22:13:10.2254899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:10.2274411Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:10.2339041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:10.2360120Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:10.2422244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:10.2442102Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:10.2503834Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:10.2532157Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:10.2598519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:10.2620012Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:10.2680886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:10.2706537Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:10.2767094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:10.2790395Z Entering 'third_party/fmt' 2024-08-20T22:13:10.2852821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:10.2872590Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:10.2936972Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:10.2957855Z Entering 'third_party/gloo' 2024-08-20T22:13:10.3020009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:10.3040718Z Entering 'third_party/googletest' 2024-08-20T22:13:10.3102396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:10.3123522Z Entering 'third_party/ideep' 2024-08-20T22:13:10.3185227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:10.3204048Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:10.3265088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:10.3294038Z Entering 'third_party/ittapi' 2024-08-20T22:13:10.3356188Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:10.3375784Z Entering 'third_party/kineto' 2024-08-20T22:13:10.3440592Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:10.3459254Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:10.3521728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:10.3540857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:10.3603648Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:10.3625454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:10.3688121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:10.3709278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:10.3771510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:10.3795857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:10.3859177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:10.3877346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:10.3940858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:10.3964876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:10.4026500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:10.4046849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:10.4109469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:10.4130157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:10.4192452Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:10.4214517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:10.4276352Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:10.4300080Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:10.4361545Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:10.4382518Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:10.4447461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:10.4470204Z Entering 'third_party/mimalloc' 2024-08-20T22:13:10.4533431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:10.4554964Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:10.4617427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:10.4638194Z Entering 'third_party/nlohmann' 2024-08-20T22:13:10.4700210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:10.4720981Z Entering 'third_party/onnx' 2024-08-20T22:13:10.4781928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:10.4815759Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:10.4881278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:10.4902675Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:10.4964556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:10.4990988Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:10.5053823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:10.5076125Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:10.5138098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:10.5158173Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:10.5220297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:10.5241437Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:10.5305071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:10.5325714Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:10.5388026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:10.5410440Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:10.5471510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:10.5491258Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:10.5554198Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:10.5576790Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:10.5638627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:10.5656272Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:10.5718873Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:10.5741077Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:10.5803771Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:10.5826778Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:10.5888078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:10.5928997Z Entering 'third_party/pocketfft' 2024-08-20T22:13:10.5990761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:10.6011963Z Entering 'third_party/protobuf' 2024-08-20T22:13:10.6074507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:10.6097232Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:10.6161161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:10.6181261Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:10.6246158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:10.6268371Z Entering 'third_party/psimd' 2024-08-20T22:13:10.6330808Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:10.6352023Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:10.6414548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:10.6435471Z Entering 'third_party/pybind11' 2024-08-20T22:13:10.6497935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:10.6519820Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:10.6583304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:10.6603946Z Entering 'third_party/sleef' 2024-08-20T22:13:10.6664491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:10.6685278Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:10.6747984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:10.6766807Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:10.6828117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:10.6848012Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:10.6912149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:10.6932857Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:10.6997759Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:10.7019052Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:10.7080700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:10.7099907Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:10.7162560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:10.8284742Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:10.8651770Z Entering 'android/libs/fbjni' 2024-08-20T22:13:10.8706827Z Entering 'third_party/FP16' 2024-08-20T22:13:10.8759812Z Entering 'third_party/FXdiv' 2024-08-20T22:13:10.8811303Z Entering 'third_party/NNPACK' 2024-08-20T22:13:10.8863003Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:10.8915274Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:10.8982437Z Entering 'third_party/benchmark' 2024-08-20T22:13:10.9034048Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:10.9085731Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:10.9138431Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:10.9190093Z Entering 'third_party/cutlass' 2024-08-20T22:13:10.9251399Z Entering 'third_party/eigen' 2024-08-20T22:13:10.9305561Z Entering 'third_party/fbgemm' 2024-08-20T22:13:10.9356597Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:10.9407568Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:10.9458958Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:10.9521655Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:10.9571635Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:10.9630503Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:10.9686406Z Entering 'third_party/fmt' 2024-08-20T22:13:10.9739968Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:10.9792427Z Entering 'third_party/gloo' 2024-08-20T22:13:10.9846195Z Entering 'third_party/googletest' 2024-08-20T22:13:10.9898977Z Entering 'third_party/ideep' 2024-08-20T22:13:10.9950016Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:11.0008180Z Entering 'third_party/ittapi' 2024-08-20T22:13:11.0059905Z Entering 'third_party/kineto' 2024-08-20T22:13:11.0110835Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:11.0160062Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:11.0212100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:11.0262261Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:11.0312657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:11.0360825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:11.0416623Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:11.0466588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:11.0517089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:11.0568192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:11.0623280Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:11.0677472Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:11.0731120Z Entering 'third_party/mimalloc' 2024-08-20T22:13:11.0783078Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:11.0835085Z Entering 'third_party/nlohmann' 2024-08-20T22:13:11.0887299Z Entering 'third_party/onnx' 2024-08-20T22:13:11.0953790Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:11.1005964Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:11.1063569Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:11.1117430Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:11.1167205Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:11.1217625Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:11.1267130Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:11.1319356Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:11.1368250Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:11.1417988Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:11.1466781Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:11.1519370Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:11.1578681Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:11.1649455Z Entering 'third_party/pocketfft' 2024-08-20T22:13:11.1704295Z Entering 'third_party/protobuf' 2024-08-20T22:13:11.1757473Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:11.1807998Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:11.1862741Z Entering 'third_party/psimd' 2024-08-20T22:13:11.1914713Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:11.1966072Z Entering 'third_party/pybind11' 2024-08-20T22:13:11.2019195Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:11.2072198Z Entering 'third_party/sleef' 2024-08-20T22:13:11.2127299Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:11.2178985Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:11.2229817Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:11.2281108Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:11.2333866Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:11.2385250Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:11.2465081Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:11.2828523Z Entering 'android/libs/fbjni' 2024-08-20T22:13:11.2881848Z Entering 'third_party/FP16' 2024-08-20T22:13:11.2939502Z Entering 'third_party/FXdiv' 2024-08-20T22:13:11.2991885Z Entering 'third_party/NNPACK' 2024-08-20T22:13:11.3045405Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:11.3098626Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:11.3166263Z Entering 'third_party/benchmark' 2024-08-20T22:13:11.3219944Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:11.3272162Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:11.3325641Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:11.3378465Z Entering 'third_party/cutlass' 2024-08-20T22:13:11.3438229Z Entering 'third_party/eigen' 2024-08-20T22:13:11.3492872Z Entering 'third_party/fbgemm' 2024-08-20T22:13:11.3545315Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:11.3598608Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:11.3651858Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:11.3712426Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:11.3762755Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:11.3816627Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:11.3873103Z Entering 'third_party/fmt' 2024-08-20T22:13:11.3926421Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:11.3979311Z Entering 'third_party/gloo' 2024-08-20T22:13:11.4032317Z Entering 'third_party/googletest' 2024-08-20T22:13:11.4085812Z Entering 'third_party/ideep' 2024-08-20T22:13:11.4136401Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:11.4194684Z Entering 'third_party/ittapi' 2024-08-20T22:13:11.4248715Z Entering 'third_party/kineto' 2024-08-20T22:13:11.4300935Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:11.4351005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:11.4403866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:11.4453735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:11.4504141Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:11.4552993Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:11.4607565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:11.4658985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:11.4710674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:11.4761259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:11.4814496Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:11.4865010Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:11.4921265Z Entering 'third_party/mimalloc' 2024-08-20T22:13:11.4974957Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:11.5028024Z Entering 'third_party/nlohmann' 2024-08-20T22:13:11.5081234Z Entering 'third_party/onnx' 2024-08-20T22:13:11.5150228Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:11.5206338Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:11.5263070Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:11.5317017Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:11.5367078Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:11.5417308Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:11.5466868Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:11.5517072Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:11.5565402Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:11.5613919Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:11.5660823Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:11.5713651Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:11.5767878Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:11.5840596Z Entering 'third_party/pocketfft' 2024-08-20T22:13:11.5892088Z Entering 'third_party/protobuf' 2024-08-20T22:13:11.5946170Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:11.5995862Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:11.6051613Z Entering 'third_party/psimd' 2024-08-20T22:13:11.6103618Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:11.6155860Z Entering 'third_party/pybind11' 2024-08-20T22:13:11.6209355Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:11.6262047Z Entering 'third_party/sleef' 2024-08-20T22:13:11.6314819Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:11.6366346Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:11.6417999Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:11.6468236Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:11.6519865Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:11.6568508Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:11.6642971Z ##[endgroup] 2024-08-20T22:13:11.6694330Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:11.6731005Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:11.6953849Z Prepare all required actions 2024-08-20T22:13:11.6954299Z Getting action download info 2024-08-20T22:13:11.8377327Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:11.8377753Z env: 2024-08-20T22:13:11.8378033Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.8378374Z ##[endgroup] 2024-08-20T22:13:11.8444061Z ##[group]Run set -euo pipefail 2024-08-20T22:13:11.8444494Z set -euo pipefail 2024-08-20T22:13:11.8444883Z function get_ec2_metadata() { 2024-08-20T22:13:11.8445411Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:11.8446286Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:11.8447040Z  category=$1 2024-08-20T22:13:11.8447583Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:11.8448219Z  runner_name_str=i-0d6323d38c97c2036 2024-08-20T22:13:11.8448716Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:11.8449205Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:11.8449768Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:11.8450451Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:11.8451056Z  else 2024-08-20T22:13:11.8451546Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:11.8452164Z  fi 2024-08-20T22:13:11.8452534Z } 2024-08-20T22:13:11.8452885Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:11.8453463Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:11.8454109Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:11.8454677Z echo "system info $(uname -a)" 2024-08-20T22:13:11.8466055Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:11.8466545Z env: 2024-08-20T22:13:11.8466827Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.8467178Z ##[endgroup] 2024-08-20T22:13:11.8574594Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:11.8631103Z instance-id: i-0d6323d38c97c2036 2024-08-20T22:13:11.8686349Z instance-type: g5.4xlarge 2024-08-20T22:13:11.8701611Z system info Linux ip-10-0-43-152.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:11.8723746Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:11.8724643Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:11.8734014Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:11.8734664Z env: 2024-08-20T22:13:11.8734944Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.8735286Z ##[endgroup] 2024-08-20T22:13:11.8853234Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:11.8853840Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:11.8854376Z  echo "Docker daemon is running..."; 2024-08-20T22:13:11.8854823Z else 2024-08-20T22:13:11.8855312Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:11.8855887Z fi 2024-08-20T22:13:11.8863946Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:11.8864437Z env: 2024-08-20T22:13:11.8864719Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.8865070Z ##[endgroup] 2024-08-20T22:13:11.8959346Z Docker daemon is running... 2024-08-20T22:13:11.9008602Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:11.9009162Z with: 2024-08-20T22:13:11.9009446Z shell: bash 2024-08-20T22:13:11.9009737Z timeout_minutes: 5 2024-08-20T22:13:11.9010060Z max_attempts: 3 2024-08-20T22:13:11.9010378Z retry_wait_seconds: 30 2024-08-20T22:13:11.9013535Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:11.9016454Z polling_interval_seconds: 1 2024-08-20T22:13:11.9016819Z warning_on_retry: true 2024-08-20T22:13:11.9017167Z continue_on_error: false 2024-08-20T22:13:11.9017509Z env: 2024-08-20T22:13:11.9017780Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.9018125Z AWS_RETRY_MODE: standard 2024-08-20T22:13:11.9018472Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:11.9018813Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:11.9019184Z ##[endgroup] 2024-08-20T22:13:13.0676842Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:13.0677900Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:13.0679011Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:13.0679685Z 2024-08-20T22:13:13.0679847Z Login Succeeded 2024-08-20T22:13:13.9583002Z Command completed after 1 attempt(s). 2024-08-20T22:13:13.9651199Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:13.9651921Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:13.9652679Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:13.9662378Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:13.9662899Z env: 2024-08-20T22:13:13.9663177Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:13.9663521Z ##[endgroup] 2024-08-20T22:13:13.9759772Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:13.9760541Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:13.9761127Z # shellcheck disable=SC2046 2024-08-20T22:13:13.9761584Z docker stop $(docker ps -q) || true 2024-08-20T22:13:13.9762060Z # Prune all of the docker images 2024-08-20T22:13:13.9762506Z docker system prune -af 2024-08-20T22:13:13.9771506Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:13.9771999Z env: 2024-08-20T22:13:13.9772273Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:13.9772747Z ##[endgroup] 2024-08-20T22:13:14.0091914Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:14.0093158Z See 'docker stop --help'. 2024-08-20T22:13:14.0093411Z 2024-08-20T22:13:14.0093637Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:14.0094007Z 2024-08-20T22:13:14.0094164Z Stop one or more running containers 2024-08-20T22:13:14.0316838Z Total reclaimed space: 0B 2024-08-20T22:13:14.0362342Z ##[group]Run set +e 2024-08-20T22:13:14.0362698Z set +e 2024-08-20T22:13:14.0363002Z set -x 2024-08-20T22:13:14.0363301Z  2024-08-20T22:13:14.0363617Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:14.0364390Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:14.0365445Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:14.0366183Z # one is returned at random 2024-08-20T22:13:14.0366722Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:14.0367229Z  2024-08-20T22:13:14.0367562Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:14.0368173Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:14.0369112Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:14.0369664Z  2024-08-20T22:13:14.0369991Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:14.0370543Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:14.0371047Z  exit 1 2024-08-20T22:13:14.0371357Z  fi 2024-08-20T22:13:14.0371645Z fi 2024-08-20T22:13:14.0371924Z  2024-08-20T22:13:14.0372304Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:14.0372948Z  # Clean up any old records first 2024-08-20T22:13:14.0373454Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:14.0373906Z fi 2024-08-20T22:13:14.0374178Z  2024-08-20T22:13:14.0374611Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:14.0375147Z cat /etc/hosts 2024-08-20T22:13:14.0384105Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:14.0384594Z env: 2024-08-20T22:13:14.0384881Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:14.0385229Z ##[endgroup] 2024-08-20T22:13:14.0415071Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:14.0421648Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:14.0422115Z ++ tail -n1 2024-08-20T22:13:14.1111431Z + RESOLVED_IP=18.160.10.28 2024-08-20T22:13:14.1111967Z + '[' -z 18.160.10.28 ']' 2024-08-20T22:13:14.1112410Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:14.1129974Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T22:13:14.1130891Z + sudo tee -a /etc/hosts 2024-08-20T22:13:14.3978569Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:14.4002376Z + cat /etc/hosts 2024-08-20T22:13:14.4014042Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:14.4026955Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:14.4027442Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:14.4178006Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:14.4178604Z with: 2024-08-20T22:13:14.4179693Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4180906Z docker-build-dir: .ci/docker 2024-08-20T22:13:14.4181288Z working-directory: . 2024-08-20T22:13:14.4181753Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.4182274Z force-push: false 2024-08-20T22:13:14.4182603Z env: 2024-08-20T22:13:14.4182900Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:14.4183240Z ##[endgroup] 2024-08-20T22:13:14.4202094Z ##[group]Run set -ex 2024-08-20T22:13:14.4202459Z set -ex 2024-08-20T22:13:14.4202749Z  2024-08-20T22:13:14.4203465Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:14.4204421Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:14.4205206Z # job could then download the pre-built image as usual 2024-08-20T22:13:14.4205939Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:14.4206600Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4207231Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4207785Z  2024-08-20T22:13:14.4208288Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:14.4208909Z  exit 0 2024-08-20T22:13:14.4209209Z else 2024-08-20T22:13:14.4209571Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4210019Z fi 2024-08-20T22:13:14.4210305Z  2024-08-20T22:13:14.4210770Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:14.4211605Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:14.4212460Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:14.4213177Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:14.4213892Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4214577Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4215123Z else 2024-08-20T22:13:14.4215555Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:14.4216212Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4217092Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4217828Z fi 2024-08-20T22:13:14.4229050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:14.4229542Z env: 2024-08-20T22:13:14.4229814Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:14.4230162Z REPO_NAME: pytorch 2024-08-20T22:13:14.4231278Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4232464Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:14.4232954Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.4233452Z ##[endgroup] 2024-08-20T22:13:14.4265642Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:14.4266291Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:14.4266761Z + echo skip=false 2024-08-20T22:13:14.4270292Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:14.4275254Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4276799Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:14.4303824Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4304906Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4306691Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4340412Z ##[group]Run set +e 2024-08-20T22:13:14.4340904Z set +e 2024-08-20T22:13:14.4341352Z set -x 2024-08-20T22:13:14.4341741Z  2024-08-20T22:13:14.4342275Z login() { 2024-08-20T22:13:14.4343109Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:14.4343875Z } 2024-08-20T22:13:14.4344267Z  2024-08-20T22:13:14.4344689Z retry () { 2024-08-20T22:13:14.4345194Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:14.4345699Z } 2024-08-20T22:13:14.4346117Z  2024-08-20T22:13:14.4346547Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:14.4347058Z  2024-08-20T22:13:14.4347637Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:14.4348449Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:14.4349042Z  exit 0 2024-08-20T22:13:14.4349479Z fi 2024-08-20T22:13:14.4349836Z  2024-08-20T22:13:14.4350505Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:14.4351490Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:14.4352321Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:14.4353043Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:14.4353792Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:14.4354318Z else 2024-08-20T22:13:14.4354939Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:14.4355703Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:14.4356297Z fi 2024-08-20T22:13:14.4356723Z  2024-08-20T22:13:14.4357154Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:14.4357711Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4358397Z  2024-08-20T22:13:14.4359130Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:14.4359972Z  exit 0 2024-08-20T22:13:14.4360393Z fi 2024-08-20T22:13:14.4360756Z  2024-08-20T22:13:14.4361283Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:14.4362388Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:14.4363232Z  exit 1 2024-08-20T22:13:14.4363625Z fi 2024-08-20T22:13:14.4364058Z  2024-08-20T22:13:14.4364626Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:14.4365707Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:14.4366698Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:14.4367735Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:14.4368930Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:14.4369826Z fi 2024-08-20T22:13:14.4370219Z  2024-08-20T22:13:14.4370667Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:14.4379256Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:14.4379808Z env: 2024-08-20T22:13:14.4380245Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:14.4380789Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:14.4381394Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:14.4382762Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4384112Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:14.4384850Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.4385563Z ##[endgroup] 2024-08-20T22:13:14.4415846Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.4416608Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.4418532Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:14.4420532Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:14.9777067Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:14.9778135Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:14.9779231Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:14.9780026Z 2024-08-20T22:13:14.9780184Z Login Succeeded 2024-08-20T22:13:14.9803951Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:15.2340360Z { 2024-08-20T22:13:15.2341243Z "schemaVersion": 2, 2024-08-20T22:13:15.2342219Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:15.2342900Z "config": { 2024-08-20T22:13:15.2343607Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:15.2344344Z "size": 48711, 2024-08-20T22:13:15.2349111Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:15.2350203Z }, 2024-08-20T22:13:15.2351155Z "layers": [ 2024-08-20T22:13:15.2352268Z { 2024-08-20T22:13:15.2353082Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2354362Z "size": 28580681, 2024-08-20T22:13:15.2355348Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:15.2356413Z }, 2024-08-20T22:13:15.2356944Z { 2024-08-20T22:13:15.2357853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2358828Z "size": 7943451, 2024-08-20T22:13:15.2359751Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:15.2360864Z }, 2024-08-20T22:13:15.2361478Z { 2024-08-20T22:13:15.2362239Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2363289Z "size": 55728572, 2024-08-20T22:13:15.2364363Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:15.2365479Z }, 2024-08-20T22:13:15.2365924Z { 2024-08-20T22:13:15.2366817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2367780Z "size": 186, 2024-08-20T22:13:15.2368907Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:15.2370050Z }, 2024-08-20T22:13:15.2370556Z { 2024-08-20T22:13:15.2371346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2372485Z "size": 6886, 2024-08-20T22:13:15.2373430Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:15.2374716Z }, 2024-08-20T22:13:15.2375503Z { 2024-08-20T22:13:15.2376360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2377298Z "size": 1329001756, 2024-08-20T22:13:15.2378300Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:15.2379460Z }, 2024-08-20T22:13:15.2379967Z { 2024-08-20T22:13:15.2380721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2381735Z "size": 62501, 2024-08-20T22:13:15.2382715Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:15.2383755Z }, 2024-08-20T22:13:15.2384326Z { 2024-08-20T22:13:15.2385101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2386093Z "size": 1684, 2024-08-20T22:13:15.2387124Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:15.2388422Z }, 2024-08-20T22:13:15.2388946Z { 2024-08-20T22:13:15.2389837Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2390743Z "size": 1523, 2024-08-20T22:13:15.2391737Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:15.2392937Z }, 2024-08-20T22:13:15.2393438Z { 2024-08-20T22:13:15.2394218Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2395307Z "size": 2528295403, 2024-08-20T22:13:15.2396278Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:15.2397341Z }, 2024-08-20T22:13:15.2398362Z { 2024-08-20T22:13:15.2399133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2399920Z "size": 86016, 2024-08-20T22:13:15.2400622Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:15.2401366Z }, 2024-08-20T22:13:15.2401702Z { 2024-08-20T22:13:15.2402285Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2402910Z "size": 1822, 2024-08-20T22:13:15.2403544Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:15.2404305Z }, 2024-08-20T22:13:15.2404666Z { 2024-08-20T22:13:15.2405150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2405875Z "size": 246761563, 2024-08-20T22:13:15.2406536Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:15.2407207Z }, 2024-08-20T22:13:15.2407618Z { 2024-08-20T22:13:15.2408148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2408833Z "size": 544, 2024-08-20T22:13:15.2409428Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:15.2410113Z }, 2024-08-20T22:13:15.2410494Z { 2024-08-20T22:13:15.2411033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2411639Z "size": 1258, 2024-08-20T22:13:15.2412412Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:15.2413133Z }, 2024-08-20T22:13:15.2413432Z { 2024-08-20T22:13:15.2414018Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2414649Z "size": 484, 2024-08-20T22:13:15.2415253Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:15.2415956Z }, 2024-08-20T22:13:15.2416290Z { 2024-08-20T22:13:15.2416838Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2417550Z "size": 91711766, 2024-08-20T22:13:15.2418153Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:15.2418859Z }, 2024-08-20T22:13:15.2419250Z { 2024-08-20T22:13:15.2419769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2420392Z "size": 3390, 2024-08-20T22:13:15.2421235Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:15.2421957Z }, 2024-08-20T22:13:15.2422253Z { 2024-08-20T22:13:15.2422824Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2423515Z "size": 1909, 2024-08-20T22:13:15.2424135Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:15.2424829Z }, 2024-08-20T22:13:15.2425186Z { 2024-08-20T22:13:15.2425707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2426410Z "size": 701, 2024-08-20T22:13:15.2427021Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:15.2427724Z }, 2024-08-20T22:13:15.2428125Z { 2024-08-20T22:13:15.2428604Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2429374Z "size": 2864254879, 2024-08-20T22:13:15.2430131Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:15.2430830Z }, 2024-08-20T22:13:15.2431130Z { 2024-08-20T22:13:15.2431714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2432346Z "size": 380, 2024-08-20T22:13:15.2433072Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:15.2433727Z }, 2024-08-20T22:13:15.2434061Z { 2024-08-20T22:13:15.2434643Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2435278Z "size": 12900, 2024-08-20T22:13:15.2435875Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:15.2436642Z }, 2024-08-20T22:13:15.2437013Z { 2024-08-20T22:13:15.2437547Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2438231Z "size": 803, 2024-08-20T22:13:15.2438812Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:15.2439517Z }, 2024-08-20T22:13:15.2439895Z { 2024-08-20T22:13:15.2440409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2441024Z "size": 106, 2024-08-20T22:13:15.2441685Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:15.2442374Z }, 2024-08-20T22:13:15.2442730Z { 2024-08-20T22:13:15.2443283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2443948Z "size": 503, 2024-08-20T22:13:15.2444567Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:15.2445304Z }, 2024-08-20T22:13:15.2445620Z { 2024-08-20T22:13:15.2446137Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2446854Z "size": 121477267, 2024-08-20T22:13:15.2447477Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:15.2448167Z }, 2024-08-20T22:13:15.2448543Z { 2024-08-20T22:13:15.2449072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2449668Z "size": 109, 2024-08-20T22:13:15.2450387Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:15.2451074Z }, 2024-08-20T22:13:15.2451448Z { 2024-08-20T22:13:15.2451949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2452698Z "size": 490, 2024-08-20T22:13:15.2453365Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:15.2454039Z }, 2024-08-20T22:13:15.2454371Z { 2024-08-20T22:13:15.2454929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2455583Z "size": 297, 2024-08-20T22:13:15.2456165Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:15.2456940Z }, 2024-08-20T22:13:15.2457374Z { 2024-08-20T22:13:15.2457891Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2458565Z "size": 103, 2024-08-20T22:13:15.2459190Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:15.2459902Z }, 2024-08-20T22:13:15.2460248Z { 2024-08-20T22:13:15.2460759Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2461419Z "size": 1473, 2024-08-20T22:13:15.2462083Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:15.2462758Z }, 2024-08-20T22:13:15.2463161Z { 2024-08-20T22:13:15.2463713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2464345Z "size": 594729516, 2024-08-20T22:13:15.2464968Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:15.2465802Z }, 2024-08-20T22:13:15.2466159Z { 2024-08-20T22:13:15.2466678Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2467316Z "size": 164, 2024-08-20T22:13:15.2467946Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:15.2468704Z }, 2024-08-20T22:13:15.2469073Z { 2024-08-20T22:13:15.2469622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2470254Z "size": 564, 2024-08-20T22:13:15.2470933Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:15.2471584Z }, 2024-08-20T22:13:15.2471923Z { 2024-08-20T22:13:15.2472516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2473205Z "size": 35874203, 2024-08-20T22:13:15.2473821Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:15.2474595Z }, 2024-08-20T22:13:15.2474929Z { 2024-08-20T22:13:15.2475435Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2476116Z "size": 104, 2024-08-20T22:13:15.2476754Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:15.2477522Z }, 2024-08-20T22:13:15.2477817Z { 2024-08-20T22:13:15.2478334Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2479034Z "size": 425, 2024-08-20T22:13:15.2479678Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:15.2480346Z }, 2024-08-20T22:13:15.2480741Z { 2024-08-20T22:13:15.2481254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2481904Z "size": 20262183, 2024-08-20T22:13:15.2482546Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:15.2483270Z }, 2024-08-20T22:13:15.2483631Z { 2024-08-20T22:13:15.2484160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2484809Z "size": 438, 2024-08-20T22:13:15.2485455Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:15.2486205Z }, 2024-08-20T22:13:15.2486511Z { 2024-08-20T22:13:15.2487057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2487745Z "size": 701, 2024-08-20T22:13:15.2488386Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:15.2489039Z }, 2024-08-20T22:13:15.2489419Z { 2024-08-20T22:13:15.2489982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2490573Z "size": 141, 2024-08-20T22:13:15.2491247Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:15.2491966Z }, 2024-08-20T22:13:15.2492298Z { 2024-08-20T22:13:15.2492987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2493646Z "size": 135, 2024-08-20T22:13:15.2494355Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:15.2495133Z }, 2024-08-20T22:13:15.2495426Z { 2024-08-20T22:13:15.2495938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2496683Z "size": 4838068375, 2024-08-20T22:13:15.2497319Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:15.2498298Z }, 2024-08-20T22:13:15.2498697Z { 2024-08-20T22:13:15.2499215Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2499832Z "size": 194, 2024-08-20T22:13:15.2500497Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:15.2501187Z }, 2024-08-20T22:13:15.2501585Z { 2024-08-20T22:13:15.2502068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2502895Z "size": 564, 2024-08-20T22:13:15.2503632Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:15.2504328Z }, 2024-08-20T22:13:15.2504654Z { 2024-08-20T22:13:15.2505227Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2505872Z "size": 43162921, 2024-08-20T22:13:15.2506718Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:15.2507426Z }, 2024-08-20T22:13:15.2507928Z { 2024-08-20T22:13:15.2508413Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2509110Z "size": 106, 2024-08-20T22:13:15.2509734Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:15.2510523Z }, 2024-08-20T22:13:15.2510955Z { 2024-08-20T22:13:15.2511541Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2512336Z "size": 1380, 2024-08-20T22:13:15.2512922Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:15.2513632Z }, 2024-08-20T22:13:15.2514018Z { 2024-08-20T22:13:15.2514500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2515164Z "size": 701, 2024-08-20T22:13:15.2515835Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:15.2516546Z }, 2024-08-20T22:13:15.2516852Z { 2024-08-20T22:13:15.2517413Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2518109Z "size": 139, 2024-08-20T22:13:15.2518772Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:15.2519427Z }, 2024-08-20T22:13:15.2519797Z { 2024-08-20T22:13:15.2520359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2521005Z "size": 118, 2024-08-20T22:13:15.2521609Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:15.2522352Z }, 2024-08-20T22:13:15.2522716Z { 2024-08-20T22:13:15.2523201Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2523902Z "size": 4324672728, 2024-08-20T22:13:15.2524624Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:15.2525333Z }, 2024-08-20T22:13:15.2525682Z { 2024-08-20T22:13:15.2526231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2526859Z "size": 174, 2024-08-20T22:13:15.2527519Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:15.2528190Z }, 2024-08-20T22:13:15.2528516Z { 2024-08-20T22:13:15.2529095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2529691Z "size": 907, 2024-08-20T22:13:15.2530336Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:15.2531130Z }, 2024-08-20T22:13:15.2531641Z { 2024-08-20T22:13:15.2532134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2532935Z "size": 701, 2024-08-20T22:13:15.2533569Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:15.2534286Z }, 2024-08-20T22:13:15.2534639Z { 2024-08-20T22:13:15.2535160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2535815Z "size": 134, 2024-08-20T22:13:15.2536494Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:15.2537160Z }, 2024-08-20T22:13:15.2537523Z { 2024-08-20T22:13:15.2538140Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2538740Z "size": 32, 2024-08-20T22:13:15.2539386Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:15.2540239Z }, 2024-08-20T22:13:15.2540587Z { 2024-08-20T22:13:15.2541069Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2541769Z "size": 158, 2024-08-20T22:13:15.2542411Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:15.2543150Z }, 2024-08-20T22:13:15.2543443Z { 2024-08-20T22:13:15.2543971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2544693Z "size": 1841, 2024-08-20T22:13:15.2545299Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:15.2545985Z }, 2024-08-20T22:13:15.2546362Z { 2024-08-20T22:13:15.2546893Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2547481Z "size": 7529776, 2024-08-20T22:13:15.2548152Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:15.2548870Z }, 2024-08-20T22:13:15.2549195Z { 2024-08-20T22:13:15.2549714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2550356Z "size": 163, 2024-08-20T22:13:15.2551015Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:15.2551758Z }, 2024-08-20T22:13:15.2552053Z { 2024-08-20T22:13:15.2552565Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2553257Z "size": 7943, 2024-08-20T22:13:15.2553843Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:15.2554531Z }, 2024-08-20T22:13:15.2554932Z { 2024-08-20T22:13:15.2555442Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2556033Z "size": 8068, 2024-08-20T22:13:15.2556724Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:15.2557408Z }, 2024-08-20T22:13:15.2557815Z { 2024-08-20T22:13:15.2558359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2558994Z "size": 302, 2024-08-20T22:13:15.2559620Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:15.2560317Z }, 2024-08-20T22:13:15.2560649Z { 2024-08-20T22:13:15.2561180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2561857Z "size": 7629638, 2024-08-20T22:13:15.2562436Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:15.2563186Z }, 2024-08-20T22:13:15.2563559Z { 2024-08-20T22:13:15.2564126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2564721Z "size": 108, 2024-08-20T22:13:15.2565391Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:15.2566117Z }, 2024-08-20T22:13:15.2566496Z { 2024-08-20T22:13:15.2566987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2567642Z "size": 54145661, 2024-08-20T22:13:15.2568436Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:15.2569132Z }, 2024-08-20T22:13:15.2569469Z { 2024-08-20T22:13:15.2570026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2570885Z "size": 473, 2024-08-20T22:13:15.2571516Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:15.2572255Z }, 2024-08-20T22:13:15.2572679Z { 2024-08-20T22:13:15.2573185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2573859Z "size": 1374858766, 2024-08-20T22:13:15.2574506Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:15.2575185Z }, 2024-08-20T22:13:15.2575554Z { 2024-08-20T22:13:15.2576069Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2576811Z "size": 106, 2024-08-20T22:13:15.2577495Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:15.2578201Z }, 2024-08-20T22:13:15.2578549Z { 2024-08-20T22:13:15.2579134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2579768Z "size": 559, 2024-08-20T22:13:15.2580366Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:15.2581107Z }, 2024-08-20T22:13:15.2581439Z { 2024-08-20T22:13:15.2581975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2582616Z "size": 46248488, 2024-08-20T22:13:15.2583273Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:15.2583967Z }, 2024-08-20T22:13:15.2584312Z { 2024-08-20T22:13:15.2584892Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2585532Z "size": 111, 2024-08-20T22:13:15.2586200Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:15.2586878Z }, 2024-08-20T22:13:15.2587219Z { 2024-08-20T22:13:15.2587780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2588427Z "size": 32, 2024-08-20T22:13:15.2589011Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:15.2589768Z }, 2024-08-20T22:13:15.2590098Z { 2024-08-20T22:13:15.2590578Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2591321Z "size": 32, 2024-08-20T22:13:15.2591939Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:15.2592676Z }, 2024-08-20T22:13:15.2593001Z { 2024-08-20T22:13:15.2593562Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2594235Z "size": 32, 2024-08-20T22:13:15.2594881Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:15.2595543Z }, 2024-08-20T22:13:15.2595946Z { 2024-08-20T22:13:15.2596457Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:15.2597084Z "size": 32, 2024-08-20T22:13:15.2598000Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:15.2598745Z } 2024-08-20T22:13:15.2599072Z ] 2024-08-20T22:13:15.2599447Z } 2024-08-20T22:13:15.2795872Z + exit 0 2024-08-20T22:13:15.2905865Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:15.2906362Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:15.2906906Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:15.2916992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:15.2917492Z env: 2024-08-20T22:13:15.2917772Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:15.2918931Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:15.2920097Z ##[endgroup] 2024-08-20T22:13:15.2953265Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:15.3009368Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:15.3009957Z with: 2024-08-20T22:13:15.3011027Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:15.3012435Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:15.3012951Z env: 2024-08-20T22:13:15.3013234Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:15.3013583Z ##[endgroup] 2024-08-20T22:13:15.3053853Z ##[group]Run set -x 2024-08-20T22:13:15.3054365Z set -x 2024-08-20T22:13:15.3054675Z set +e 2024-08-20T22:13:15.3054972Z  2024-08-20T22:13:15.3055249Z login() { 2024-08-20T22:13:15.3055906Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:15.3056621Z } 2024-08-20T22:13:15.3056906Z  2024-08-20T22:13:15.3057216Z retry () { 2024-08-20T22:13:15.3057610Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:15.3058052Z } 2024-08-20T22:13:15.3058339Z  2024-08-20T22:13:15.3058665Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:15.3059077Z  2024-08-20T22:13:15.3059355Z set -e 2024-08-20T22:13:15.3059834Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:15.3060535Z # only pull docker image if it's not available locally 2024-08-20T22:13:15.3061293Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:15.3061991Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:15.3062434Z fi 2024-08-20T22:13:15.3071182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:15.3071678Z env: 2024-08-20T22:13:15.3071961Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:15.3073105Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:15.3074394Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:15.3074904Z ##[endgroup] 2024-08-20T22:13:15.3104676Z + set +e 2024-08-20T22:13:15.3105348Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:15.3105995Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:15.3109298Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:15.3110352Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:15.8408108Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:15.8409044Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:15.8410138Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:15.8410681Z 2024-08-20T22:13:15.8410813Z Login Succeeded 2024-08-20T22:13:15.8431068Z + set -e 2024-08-20T22:13:15.8432676Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:15.8586631Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:15.8588779Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:16.1482627Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:16.1483691Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:16.1484213Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:16.1486459Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:16.1486995Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:16.1487473Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:16.1487968Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:16.1488455Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:16.1488931Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:16.1489389Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:16.1489884Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:16.1490379Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:16.1490885Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:16.1491369Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:16.1492070Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:16.1492675Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:16.1493159Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:16.1493648Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:16.1494190Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:16.1494677Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:16.1495075Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:16.1495443Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:16.1495816Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:16.1496192Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:16.1496559Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:16.1496916Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:16.1497284Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:16.1497937Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:16.1498363Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:16.1498726Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:16.1499096Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:16.1499471Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:16.1499831Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:16.1500228Z a8c1e85b5e14: Waiting 2024-08-20T22:13:16.1500562Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:16.1500917Z 7ed32bc8e469: Waiting 2024-08-20T22:13:16.1501241Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:16.1501605Z ca426ffc9d2b: Waiting 2024-08-20T22:13:16.1501943Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:16.1502309Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:16.1502675Z 019620471c25: Pulling fs layer 2024-08-20T22:13:16.1503094Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:16.1503470Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:16.1503818Z a41a8d1c11c8: Waiting 2024-08-20T22:13:16.1504152Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:16.1504504Z 0c1227890755: Waiting 2024-08-20T22:13:16.1504830Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:16.1505178Z 75b2abb16c68: Waiting 2024-08-20T22:13:16.1505516Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:16.1505881Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:16.1506208Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:16.1506570Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:16.1506907Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:16.1507266Z 0bf61114d37b: Waiting 2024-08-20T22:13:16.1507576Z b57676e46aee: Waiting 2024-08-20T22:13:16.1507898Z d527cbbb87e3: Waiting 2024-08-20T22:13:16.1508238Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:16.1508597Z 34c47d63c5de: Waiting 2024-08-20T22:13:16.1508919Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:16.1509284Z ec1e7978c1fe: Waiting 2024-08-20T22:13:16.1509620Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:16.1509987Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:16.1510346Z 1fef4bd19792: Waiting 2024-08-20T22:13:16.1510663Z 4373aa743caf: Waiting 2024-08-20T22:13:16.1510981Z 560cd6122a38: Waiting 2024-08-20T22:13:16.1511301Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:16.1511675Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:16.1512057Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:16.1512474Z f957c1624481: Pulling fs layer 2024-08-20T22:13:16.1512836Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:16.1513233Z 5a4eb19d7219: Waiting 2024-08-20T22:13:16.1513586Z b119c4fc6c58: Waiting 2024-08-20T22:13:16.1513912Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:16.1514277Z da6a838a66c7: Waiting 2024-08-20T22:13:16.1514804Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:16.1515175Z 33b639226eee: Waiting 2024-08-20T22:13:16.1515501Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:16.1515856Z 6efb96fc87ea: Waiting 2024-08-20T22:13:16.1516185Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:16.1516546Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:16.1516905Z 019620471c25: Waiting 2024-08-20T22:13:16.1517228Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:16.1517600Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:16.1517953Z 7cbfce999d76: Waiting 2024-08-20T22:13:16.1518291Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:16.1518645Z 6658e558ebe4: Waiting 2024-08-20T22:13:16.1519103Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:16.1519467Z ee1273153675: Pulling fs layer 2024-08-20T22:13:16.1519833Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:16.1520205Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:16.1520553Z 7754ee068b33: Waiting 2024-08-20T22:13:16.1520883Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:16.1521263Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:16.1521631Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:16.1521982Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:16.1522299Z afc429e87b59: Waiting 2024-08-20T22:13:16.1522630Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:16.1523004Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:16.1523368Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:16.1523718Z 5645520256fa: Waiting 2024-08-20T22:13:16.1524027Z 1811ff4d3556: Waiting 2024-08-20T22:13:16.1524331Z 829c0e19f4b7: Waiting 2024-08-20T22:13:16.1524641Z d3547bd43146: Waiting 2024-08-20T22:13:16.1524954Z 4f4fb700ef54: Waiting 2024-08-20T22:13:16.1525278Z 829ed7d91e38: Waiting 2024-08-20T22:13:16.1525587Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:16.1525902Z 78ee77eddef2: Waiting 2024-08-20T22:13:16.1526212Z 33444a5a7e01: Waiting 2024-08-20T22:13:16.1526514Z 916c5e858016: Waiting 2024-08-20T22:13:16.1526825Z c42b3602a1fb: Waiting 2024-08-20T22:13:16.1527142Z f1d6874dee9c: Waiting 2024-08-20T22:13:16.1527459Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:16.1527762Z 7c47122f4e09: Waiting 2024-08-20T22:13:16.1528070Z 385d0421d67c: Waiting 2024-08-20T22:13:16.1528381Z 11fad972ef6c: Waiting 2024-08-20T22:13:16.1528696Z eb09cbf3e416: Waiting 2024-08-20T22:13:16.1529001Z 5d115b29b5d5: Waiting 2024-08-20T22:13:16.1529309Z e22979f9e0ee: Waiting 2024-08-20T22:13:16.1529624Z b10de02255c0: Waiting 2024-08-20T22:13:16.1529925Z d79fd7d52698: Waiting 2024-08-20T22:13:16.1530238Z e56eeff2db93: Waiting 2024-08-20T22:13:16.1530553Z 37513e9b43d7: Waiting 2024-08-20T22:13:16.1530863Z 768c31dca016: Waiting 2024-08-20T22:13:16.1531168Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:16.1531484Z ee1273153675: Waiting 2024-08-20T22:13:16.1531794Z a7a5392e2bf6: Waiting 2024-08-20T22:13:16.1532099Z 4a0412373305: Waiting 2024-08-20T22:13:16.1532512Z 02970347dff6: Waiting 2024-08-20T22:13:16.1532829Z 83808b2eff4d: Waiting 2024-08-20T22:13:16.1533183Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:16.1533486Z 302c8c177944: Waiting 2024-08-20T22:13:16.1533801Z 9752d6fbbcab: Waiting 2024-08-20T22:13:16.1534112Z f957c1624481: Waiting 2024-08-20T22:13:16.1534423Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:16.2794308Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:16.2794893Z 224fe954d725: Download complete 2024-08-20T22:13:16.3563620Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:16.3564207Z d527cbbb87e3: Download complete 2024-08-20T22:13:16.4248328Z b57676e46aee: Download complete 2024-08-20T22:13:16.4909313Z 7a2c55901189: Download complete 2024-08-20T22:13:16.5549374Z a41a8d1c11c8: Download complete 2024-08-20T22:13:16.6242361Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:16.6242915Z 0c1227890755: Download complete 2024-08-20T22:13:16.6843079Z d8d1234baab3: Verifying Checksum 2024-08-20T22:13:16.6843834Z d8d1234baab3: Download complete 2024-08-20T22:13:16.7610758Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:16.7611184Z 75722010b82e: Download complete 2024-08-20T22:13:16.8334999Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:16.8336080Z ec1e7978c1fe: Download complete 2024-08-20T22:13:16.9039844Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:13:16.9040331Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:17.6825725Z 7a2c55901189: Pull complete 2024-08-20T22:13:18.0063085Z 224fe954d725: Pull complete 2024-08-20T22:13:18.9370531Z 75722010b82e: Pull complete 2024-08-20T22:13:18.9648010Z d527cbbb87e3: Pull complete 2024-08-20T22:13:18.9871192Z b57676e46aee: Pull complete 2024-08-20T22:13:19.4316838Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:19.4317380Z ca426ffc9d2b: Download complete 2024-08-20T22:13:19.5102006Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:19.5102944Z 4373aa743caf: Download complete 2024-08-20T22:13:19.5938857Z 75b2abb16c68: Download complete 2024-08-20T22:13:19.6652533Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:13:19.6653245Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:20.6478330Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:20.6478829Z 0bf61114d37b: Download complete 2024-08-20T22:13:20.7240434Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:20.7240952Z 2df849f72ba4: Download complete 2024-08-20T22:13:20.8052090Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:20.8052851Z 34c47d63c5de: Download complete 2024-08-20T22:13:20.8961171Z 1fef4bd19792: Download complete 2024-08-20T22:13:29.8089582Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:29.8090101Z a8c1e85b5e14: Download complete 2024-08-20T22:13:29.8890542Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:29.8891068Z b119c4fc6c58: Download complete 2024-08-20T22:13:29.9728356Z 33b639226eee: Verifying Checksum 2024-08-20T22:13:29.9728837Z 33b639226eee: Download complete 2024-08-20T22:13:30.0384916Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:30.0385375Z d3547bd43146: Download complete 2024-08-20T22:13:30.1106779Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:30.1107389Z 5645520256fa: Download complete 2024-08-20T22:13:30.1903252Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:13:30.1903930Z 829ed7d91e38: Download complete 2024-08-20T22:13:31.4678526Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:31.4679156Z 385d0421d67c: Download complete 2024-08-20T22:13:31.5499998Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:31.5500457Z 7cbfce999d76: Download complete 2024-08-20T22:13:31.6219905Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:31.6220360Z 6658e558ebe4: Download complete 2024-08-20T22:13:31.7072860Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:13:31.7073328Z 33444a5a7e01: Download complete 2024-08-20T22:13:31.7838507Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:31.7838962Z 7c47122f4e09: Download complete 2024-08-20T22:13:31.8661625Z 1811ff4d3556: Download complete 2024-08-20T22:13:37.8801629Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:37.9445478Z 0479fbac3361: Verifying Checksum 2024-08-20T22:13:37.9446062Z 0479fbac3361: Download complete 2024-08-20T22:13:38.0309326Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:13:38.0310012Z f1d6874dee9c: Download complete 2024-08-20T22:13:38.4476298Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:38.4476786Z afc429e87b59: Download complete 2024-08-20T22:13:38.5262442Z 019620471c25: Verifying Checksum 2024-08-20T22:13:38.5263010Z 019620471c25: Download complete 2024-08-20T22:13:38.6028136Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:38.6028784Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:38.8561557Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:38.8562160Z 829c0e19f4b7: Download complete 2024-08-20T22:13:38.9200186Z 560cd6122a38: Download complete 2024-08-20T22:13:38.9887744Z 5a4eb19d7219: Download complete 2024-08-20T22:13:39.0674262Z da6a838a66c7: Verifying Checksum 2024-08-20T22:13:39.0674950Z da6a838a66c7: Download complete 2024-08-20T22:13:42.0220294Z 7ed32bc8e469: Download complete 2024-08-20T22:13:42.0935365Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:13:42.0936022Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:42.1759389Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:13:42.1759855Z 0f085d4fa279: Download complete 2024-08-20T22:13:42.6524598Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:42.6525077Z 4a0412373305: Download complete 2024-08-20T22:13:42.7473505Z 302c8c177944: Verifying Checksum 2024-08-20T22:13:42.7473986Z 302c8c177944: Download complete 2024-08-20T22:13:42.8350996Z 02970347dff6: Verifying Checksum 2024-08-20T22:13:42.8351447Z 02970347dff6: Download complete 2024-08-20T22:13:42.9043482Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:42.9044006Z 916c5e858016: Download complete 2024-08-20T22:13:42.9709587Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:42.9710195Z c42b3602a1fb: Download complete 2024-08-20T22:13:43.1071150Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:43.3146206Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:43.5091837Z 0c1227890755: Pull complete 2024-08-20T22:13:43.6593558Z d8d1234baab3: Pull complete 2024-08-20T22:13:49.5872424Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:49.5873083Z 6efb96fc87ea: Download complete 2024-08-20T22:13:49.6563263Z f957c1624481: Verifying Checksum 2024-08-20T22:13:49.6563867Z f957c1624481: Download complete 2024-08-20T22:13:49.7263499Z 768c31dca016: Download complete 2024-08-20T22:13:49.7964810Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:13:49.7965389Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:49.8061489Z 4f4fb700ef54: Download complete 2024-08-20T22:13:49.8824036Z 78ee77eddef2: Download complete 2024-08-20T22:13:49.9569166Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:13:49.9569634Z 37513e9b43d7: Download complete 2024-08-20T22:13:50.0801340Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:50.0801944Z a7a5392e2bf6: Download complete 2024-08-20T22:13:50.1586092Z b10de02255c0: Download complete 2024-08-20T22:13:50.2164647Z e56eeff2db93: Download complete 2024-08-20T22:13:50.2852223Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:13:50.2852719Z 70ac3e38d0fd: Download complete 2024-08-20T22:13:50.3556358Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:13:50.3556851Z eb09cbf3e416: Download complete 2024-08-20T22:13:50.4825919Z ee1273153675: Verifying Checksum 2024-08-20T22:13:50.4826604Z ee1273153675: Download complete 2024-08-20T22:13:50.5576962Z e22979f9e0ee: Download complete 2024-08-20T22:13:51.1478656Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:51.1479227Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:51.2201836Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:13:51.2202405Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:16.6402715Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:16.6403191Z 5d115b29b5d5: Download complete 2024-08-20T22:14:16.7148693Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:16.7149273Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:16.7842616Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:16.7843064Z d79fd7d52698: Download complete 2024-08-20T22:14:17.9903960Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:17.9904446Z 9752d6fbbcab: Download complete 2024-08-20T22:14:18.0516858Z 83808b2eff4d: Download complete 2024-08-20T22:14:57.1405130Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:14:57.1405703Z 11fad972ef6c: Download complete 2024-08-20T22:14:58.3398474Z cbb79a705353: Verifying Checksum 2024-08-20T22:15:12.3768266Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:12.6031464Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:12.8271628Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:21.4127896Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:21.6010162Z 4373aa743caf: Pull complete 2024-08-20T22:15:21.8248642Z 75b2abb16c68: Pull complete 2024-08-20T22:15:22.0296017Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:24.5867972Z 0bf61114d37b: Pull complete 2024-08-20T22:15:24.6172914Z 2df849f72ba4: Pull complete 2024-08-20T22:15:24.6686070Z 34c47d63c5de: Pull complete 2024-08-20T22:15:24.7364570Z 1fef4bd19792: Pull complete 2024-08-20T22:16:22.4858314Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:22.6353584Z b119c4fc6c58: Pull complete 2024-08-20T22:16:22.7958598Z 33b639226eee: Pull complete 2024-08-20T22:16:22.9519880Z d3547bd43146: Pull complete 2024-08-20T22:16:23.0969791Z 5645520256fa: Pull complete 2024-08-20T22:16:23.2717008Z 829ed7d91e38: Pull complete 2024-08-20T22:16:26.6196760Z 385d0421d67c: Pull complete 2024-08-20T22:16:26.7600136Z 7cbfce999d76: Pull complete 2024-08-20T22:16:26.9143543Z 6658e558ebe4: Pull complete 2024-08-20T22:16:27.0688857Z 33444a5a7e01: Pull complete 2024-08-20T22:16:27.1489017Z 7c47122f4e09: Pull complete 2024-08-20T22:16:27.3608642Z 1811ff4d3556: Pull complete 2024-08-20T22:16:37.9139413Z 7754ee068b33: Pull complete 2024-08-20T22:16:38.1236981Z 0479fbac3361: Pull complete 2024-08-20T22:16:38.2168714Z f1d6874dee9c: Pull complete 2024-08-20T22:16:39.1439421Z afc429e87b59: Pull complete 2024-08-20T22:16:39.3548637Z 019620471c25: Pull complete 2024-08-20T22:16:39.5484208Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:40.0328985Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:40.1977470Z 560cd6122a38: Pull complete 2024-08-20T22:16:40.5162220Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:40.7102672Z da6a838a66c7: Pull complete 2024-08-20T22:17:19.3531972Z cbb79a705353: Pull complete 2024-08-20T22:17:19.5788399Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:19.6977414Z 0f085d4fa279: Pull complete 2024-08-20T22:17:22.0201376Z 4a0412373305: Pull complete 2024-08-20T22:17:22.0444550Z 302c8c177944: Pull complete 2024-08-20T22:17:22.0687367Z 02970347dff6: Pull complete 2024-08-20T22:17:22.1075189Z 916c5e858016: Pull complete 2024-08-20T22:17:22.1263372Z c42b3602a1fb: Pull complete 2024-08-20T22:18:43.4784009Z 11fad972ef6c: Pull complete 2024-08-20T22:18:43.6894866Z f957c1624481: Pull complete 2024-08-20T22:18:43.9071550Z 768c31dca016: Pull complete 2024-08-20T22:18:44.3251201Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:44.5454574Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:44.7601427Z 78ee77eddef2: Pull complete 2024-08-20T22:18:44.9823102Z 37513e9b43d7: Pull complete 2024-08-20T22:18:45.3213388Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:45.5121741Z b10de02255c0: Pull complete 2024-08-20T22:18:45.7343457Z e56eeff2db93: Pull complete 2024-08-20T22:18:45.9510785Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:46.1744879Z eb09cbf3e416: Pull complete 2024-08-20T22:18:47.5703603Z ee1273153675: Pull complete 2024-08-20T22:18:47.7924102Z e22979f9e0ee: Pull complete 2024-08-20T22:18:49.9681778Z 6c1f0aabce7c: Pull complete 2024-08-20T22:18:50.1474609Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:03.4883058Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:03.6921592Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:03.8884962Z d79fd7d52698: Pull complete 2024-08-20T22:19:04.7461879Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:04.9534371Z 83808b2eff4d: Pull complete 2024-08-20T22:19:05.8460237Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:05.8941921Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:05.9142398Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:05.9248957Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:05.9249886Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:05.9261173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:05.9261675Z env: 2024-08-20T22:19:05.9261955Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:05.9262308Z ##[endgroup] 2024-08-20T22:19:05.9485510Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:05.9486064Z with: 2024-08-20T22:19:05.9486361Z driver-version: 550.54.15 2024-08-20T22:19:05.9486699Z env: 2024-08-20T22:19:05.9486973Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:05.9487320Z ##[endgroup] 2024-08-20T22:19:05.9595630Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:05.9596217Z with: 2024-08-20T22:19:05.9596515Z timeout_minutes: 10 2024-08-20T22:19:05.9596836Z max_attempts: 3 2024-08-20T22:19:05.9629206Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:05.9661777Z retry_wait_seconds: 10 2024-08-20T22:19:05.9662143Z polling_interval_seconds: 1 2024-08-20T22:19:05.9662510Z warning_on_retry: true 2024-08-20T22:19:05.9662865Z continue_on_error: false 2024-08-20T22:19:05.9663196Z env: 2024-08-20T22:19:05.9663465Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:05.9663806Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:05.9664153Z ##[endgroup] 2024-08-20T22:19:06.0466495Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:06.0467632Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:06.0468235Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:06.5081826Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:06.5084057Z No packages marked for removal. 2024-08-20T22:19:06.5146098Z Dependencies resolved. 2024-08-20T22:19:06.5157298Z Nothing to do. 2024-08-20T22:19:06.5158119Z Complete! 2024-08-20T22:19:06.5999265Z + install_nvidia_driver_common 2024-08-20T22:19:06.6002524Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:06.6002951Z + lspci 2024-08-20T22:19:06.6003962Z Before installing NVIDIA driver 2024-08-20T22:19:06.6887075Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:06.6887856Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:06.6888781Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:06.6889587Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:06.6890402Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:06.6891195Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:06.6891935Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:06.6892817Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:06.6893404Z + lsmod 2024-08-20T22:19:06.6933747Z Module Size Used by 2024-08-20T22:19:06.6934191Z xt_conntrack 16384 1 2024-08-20T22:19:06.6934570Z nft_chain_nat 16384 3 2024-08-20T22:19:06.6934968Z xt_MASQUERADE 20480 1 2024-08-20T22:19:06.6935427Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:06.6935925Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:06.6936505Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:06.6937150Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:06.6937629Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:06.6938068Z xfrm_user 57344 1 2024-08-20T22:19:06.6938459Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:06.6938890Z xt_addrtype 16384 2 2024-08-20T22:19:06.6939271Z nft_compat 20480 4 2024-08-20T22:19:06.6939720Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:06.6941770Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:06.6942442Z br_netfilter 36864 0 2024-08-20T22:19:06.6942990Z bridge 307200 1 br_netfilter 2024-08-20T22:19:06.6943798Z stp 16384 1 bridge 2024-08-20T22:19:06.6944306Z llc 16384 2 bridge,stp 2024-08-20T22:19:06.6944727Z overlay 167936 0 2024-08-20T22:19:06.6945091Z tls 114688 0 2024-08-20T22:19:06.6945458Z nls_ascii 16384 1 2024-08-20T22:19:06.6945824Z nls_cp437 20480 1 2024-08-20T22:19:06.6946192Z vfat 24576 1 2024-08-20T22:19:06.6946597Z fat 86016 1 vfat 2024-08-20T22:19:06.6946985Z sunrpc 692224 1 2024-08-20T22:19:06.6947342Z ena 167936 0 2024-08-20T22:19:06.6947716Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:06.6948087Z ptp 36864 1 ena 2024-08-20T22:19:06.6948480Z pps_core 24576 1 ptp 2024-08-20T22:19:06.6948875Z aesni_intel 393216 0 2024-08-20T22:19:06.6949234Z i8042 45056 0 2024-08-20T22:19:06.6949615Z serio 28672 3 i8042 2024-08-20T22:19:06.6950045Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:06.6950561Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:06.6951054Z button 24576 0 2024-08-20T22:19:06.6951422Z sch_fq_codel 20480 17 2024-08-20T22:19:06.6951795Z dm_mod 188416 0 2024-08-20T22:19:06.6952161Z fuse 163840 1 2024-08-20T22:19:06.6952521Z configfs 57344 1 2024-08-20T22:19:06.6952894Z dax 45056 1 dm_mod 2024-08-20T22:19:06.6953286Z loop 36864 0 2024-08-20T22:19:06.6953651Z dmi_sysfs 20480 0 2024-08-20T22:19:06.6954013Z crc32_pclmul 16384 0 2024-08-20T22:19:06.6954383Z crc32c_intel 24576 0 2024-08-20T22:19:06.6954749Z efivarfs 24576 1 2024-08-20T22:19:06.6955110Z + modinfo nvidia 2024-08-20T22:19:06.6955971Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:06.6956753Z alias: char-major-195-* 2024-08-20T22:19:06.6957182Z version: 550.54.15 2024-08-20T22:19:06.6957525Z supported: external 2024-08-20T22:19:06.6957858Z license: NVIDIA 2024-08-20T22:19:06.6958252Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:06.6958735Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:06.6959196Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:06.6959651Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:06.6960143Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:06.6960709Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:06.6961366Z depends: i2c-core,drm 2024-08-20T22:19:06.6961732Z retpoline: Y 2024-08-20T22:19:06.6962039Z name: nvidia 2024-08-20T22:19:06.6962798Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:06.6963493Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:06.6964147Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:06.6964734Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:06.6965176Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:06.6965602Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:06.6966054Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:06.6966487Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:06.6966921Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:06.6967475Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:06.6968027Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:06.6968501Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:06.6968931Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:06.6969345Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:06.6969802Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:06.6970328Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:06.6970995Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:06.6971544Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:06.6972145Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:06.6972868Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:06.6973474Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:06.6974054Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:06.6974546Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:06.6975077Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:06.6975619Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:06.6976104Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:06.6976567Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:06.6977044Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:06.6977519Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:06.6977957Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:06.6978456Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:06.6978975Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:06.6979448Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:06.6979921Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:06.6980413Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:06.6980894Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:06.6981382Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:06.6981855Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:06.6982263Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:06.6982722Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:06.6983185Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:06.6983638Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:06.6984116Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:06.6984588Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:06.6985028Z parm: rm_firmware_active:charp 2024-08-20T22:19:06.6985443Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:06.6985825Z ++ command -v nvidia-smi 2024-08-20T22:19:06.6986227Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:06.6986634Z + set +e 2024-08-20T22:19:06.6987146Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:08.9511109Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:08.9511684Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:08.9512272Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:08.9512653Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:08.9513036Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:08.9514146Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:08.9514860Z + set -e 2024-08-20T22:19:08.9515174Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:08.9515764Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:08.9516457Z + post_install_nvidia_driver_common 2024-08-20T22:19:08.9517411Z + sudo modprobe nvidia 2024-08-20T22:19:09.0695025Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:09.0695540Z After installing NVIDIA driver 2024-08-20T22:19:09.0695911Z + lspci 2024-08-20T22:19:09.0806477Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:09.0807578Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:09.0808513Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:09.0809300Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:09.0810149Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:09.0810943Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:09.0812067Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:09.0812988Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:09.0813568Z + lsmod 2024-08-20T22:19:09.0842431Z Module Size Used by 2024-08-20T22:19:09.0842948Z nvidia_uvm 4706304 0 2024-08-20T22:19:09.0843480Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:19:09.0844042Z drm 602112 1 nvidia 2024-08-20T22:19:09.0844625Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:09.0845226Z backlight 24576 1 drm 2024-08-20T22:19:09.0845771Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:09.0846337Z xt_conntrack 16384 1 2024-08-20T22:19:09.0846857Z nft_chain_nat 16384 3 2024-08-20T22:19:09.0847407Z xt_MASQUERADE 20480 1 2024-08-20T22:19:09.0847919Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:09.0848411Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:09.0849012Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:09.0849655Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:09.0850126Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:09.0850558Z xfrm_user 57344 1 2024-08-20T22:19:09.0850965Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:09.0851388Z xt_addrtype 16384 2 2024-08-20T22:19:09.0851765Z nft_compat 20480 4 2024-08-20T22:19:09.0852233Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:09.0852957Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:09.0853509Z br_netfilter 36864 0 2024-08-20T22:19:09.0853919Z bridge 307200 1 br_netfilter 2024-08-20T22:19:09.0854373Z stp 16384 1 bridge 2024-08-20T22:19:09.0854798Z llc 16384 2 bridge,stp 2024-08-20T22:19:09.0855221Z overlay 167936 0 2024-08-20T22:19:09.0855595Z tls 114688 0 2024-08-20T22:19:09.0855967Z nls_ascii 16384 1 2024-08-20T22:19:09.0856340Z nls_cp437 20480 1 2024-08-20T22:19:09.0856707Z vfat 24576 1 2024-08-20T22:19:09.0857076Z fat 86016 1 vfat 2024-08-20T22:19:09.0857482Z sunrpc 692224 1 2024-08-20T22:19:09.0857852Z ena 167936 0 2024-08-20T22:19:09.0858219Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:09.0858600Z ptp 36864 1 ena 2024-08-20T22:19:09.0858998Z pps_core 24576 1 ptp 2024-08-20T22:19:09.0859403Z aesni_intel 393216 0 2024-08-20T22:19:09.0859801Z i8042 45056 0 2024-08-20T22:19:09.0860305Z serio 28672 3 i8042 2024-08-20T22:19:09.0861182Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:09.0861893Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:09.0862548Z button 24576 0 2024-08-20T22:19:09.0863050Z sch_fq_codel 20480 17 2024-08-20T22:19:09.0863519Z dm_mod 188416 0 2024-08-20T22:19:09.0863990Z fuse 163840 1 2024-08-20T22:19:09.0864363Z configfs 57344 1 2024-08-20T22:19:09.0864834Z dax 45056 1 dm_mod 2024-08-20T22:19:09.0865356Z loop 36864 0 2024-08-20T22:19:09.0865838Z dmi_sysfs 20480 0 2024-08-20T22:19:09.0866336Z crc32_pclmul 16384 0 2024-08-20T22:19:09.0866826Z crc32c_intel 24576 0 2024-08-20T22:19:09.0867346Z efivarfs 24576 1 2024-08-20T22:19:09.0867712Z + modinfo nvidia 2024-08-20T22:19:09.0868430Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:09.0869140Z alias: char-major-195-* 2024-08-20T22:19:09.0869528Z version: 550.54.15 2024-08-20T22:19:09.0869872Z supported: external 2024-08-20T22:19:09.0870217Z license: NVIDIA 2024-08-20T22:19:09.0870754Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:09.0871245Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:09.0871702Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:09.0872174Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:09.0872681Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:09.0873172Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:09.0873663Z depends: i2c-core,drm 2024-08-20T22:19:09.0874030Z retpoline: Y 2024-08-20T22:19:09.0874348Z name: nvidia 2024-08-20T22:19:09.0874973Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:09.0875672Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:09.0876326Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:09.0876921Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:09.0877379Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:09.0877811Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:09.0878271Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:09.0878714Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:09.0879156Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:09.0879689Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:09.0880250Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:09.0880730Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:09.0881157Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:09.0881584Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:09.0882045Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:09.0882579Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:09.0883162Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:09.0883726Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:09.0884345Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:09.0884942Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:09.0885546Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:09.0886144Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:09.0886635Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:09.0887165Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:09.0887710Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:09.0888217Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:09.0888680Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:09.0889170Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:09.0889642Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:09.0890191Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:09.0890700Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:09.0891225Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:09.0891713Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:09.0892205Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:09.0892860Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:09.0893343Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:09.0893843Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:09.0894323Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:09.0894739Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:09.0895208Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:09.0895685Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:09.0896145Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:09.0896634Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:09.0897105Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:09.0897908Z parm: rm_firmware_active:charp 2024-08-20T22:19:09.0898476Z + set +e 2024-08-20T22:19:09.0898796Z + nvidia-smi 2024-08-20T22:19:10.6570582Z Tue Aug 20 22:19:10 2024 2024-08-20T22:19:10.6571583Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:10.6572556Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:10.6573347Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:10.6574155Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:10.6575038Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:10.6575729Z | | | MIG M. | 2024-08-20T22:19:10.6576299Z |=========================================+========================+======================| 2024-08-20T22:19:10.6671515Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:10.6672481Z | 0% 28C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:10.6673316Z | | | N/A | 2024-08-20T22:19:10.6674166Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:10.6674756Z 2024-08-20T22:19:10.6675411Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:10.6676040Z | Processes: | 2024-08-20T22:19:10.6676787Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:10.6677690Z | ID ID Usage | 2024-08-20T22:19:10.6678313Z |=========================================================================================| 2024-08-20T22:19:10.6678958Z | No running processes found | 2024-08-20T22:19:10.6679733Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:11.2945061Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:12.8666108Z NVIDIA A10G 2024-08-20T22:19:13.3246806Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:13.3247455Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:13.3247953Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:13.3248777Z + set -e 2024-08-20T22:19:13.3249336Z INFO: Ignoring allowed status 0 2024-08-20T22:19:13.3257816Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:13.3262688Z + sudo yum install -y yum-utils 2024-08-20T22:19:13.7841418Z Last metadata expiration check: 0:09:34 ago on Tue Aug 20 22:09:39 2024. 2024-08-20T22:19:13.8061385Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:13.8359530Z Dependencies resolved. 2024-08-20T22:19:13.8484784Z Nothing to do. 2024-08-20T22:19:13.8485108Z Complete! 2024-08-20T22:19:13.9298838Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:13.9299887Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:13.9301376Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:14.1753727Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:14.2479915Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:14.7353917Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:19:14.7576353Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:14.7879777Z Dependencies resolved. 2024-08-20T22:19:14.8007184Z Nothing to do. 2024-08-20T22:19:14.8007567Z Complete! 2024-08-20T22:19:14.8830644Z + sudo systemctl restart docker 2024-08-20T22:20:10.0047135Z Tue Aug 20 22:20:09 2024 2024-08-20T22:20:10.0048018Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:10.0048861Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:10.0049661Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:10.0050455Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:10.0051348Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:10.0055756Z | | | MIG M. | 2024-08-20T22:20:10.0056394Z |=========================================+========================+======================| 2024-08-20T22:20:10.0163970Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:10.0164704Z | 0% 29C P0 59W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:20:10.0165349Z | | | N/A | 2024-08-20T22:20:10.0166688Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:10.0167864Z 2024-08-20T22:20:10.0169190Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:10.0170449Z | Processes: | 2024-08-20T22:20:10.0171869Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:10.0173401Z | ID ID Usage | 2024-08-20T22:20:10.0174525Z |=========================================================================================| 2024-08-20T22:20:10.0175680Z | No running processes found | 2024-08-20T22:20:10.0176450Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:11.0872894Z Command completed after 1 attempt(s). 2024-08-20T22:20:11.0954777Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:11.0955561Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:11.0956254Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:11.0956914Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:11.0971050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:11.0971570Z env: 2024-08-20T22:20:11.0971863Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:11.0972403Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:11.0972900Z ##[endgroup] 2024-08-20T22:20:11.5087369Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:11.9330947Z Collecting psutil==5.9.1 2024-08-20T22:20:11.9677902Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:12.0216520Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:12.0246358Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:12.1149415Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:12.2734058Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:12.4803747Z Prepare all required actions 2024-08-20T22:20:12.4804245Z Getting action download info 2024-08-20T22:20:12.5738304Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:12.7446695Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:12.8805573Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:12.8806050Z with: 2024-08-20T22:20:12.8806389Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:12.8806846Z s3-bucket: gha-artifacts 2024-08-20T22:20:12.8807184Z env: 2024-08-20T22:20:12.8807455Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:12.8807910Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:12.8808390Z ##[endgroup] 2024-08-20T22:20:12.8842291Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:12.8842740Z with: 2024-08-20T22:20:12.8843108Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:12.8843558Z s3-bucket: gha-artifacts 2024-08-20T22:20:12.8843920Z region: us-east-1 2024-08-20T22:20:12.8844226Z env: 2024-08-20T22:20:12.8844513Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:12.8844969Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:12.8845458Z ##[endgroup] 2024-08-20T22:20:13.3603823Z (node:52496) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:13.3604731Z 2024-08-20T22:20:13.3605082Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:13.3605836Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:13.3606727Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:13.4176751Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:13.4177896Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0022311Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0028230Z Artifact download has finished successfully 2024-08-20T22:20:25.0392128Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:25.0392574Z unzip -o artifacts.zip 2024-08-20T22:20:25.0401802Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:25.0402313Z env: 2024-08-20T22:20:25.0402593Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:25.0403053Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:25.0403624Z ##[endgroup] 2024-08-20T22:20:25.0741979Z Archive: artifacts.zip 2024-08-20T22:20:25.0743397Z creating: dist/ 2024-08-20T22:20:27.2055126Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:27.2055817Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:27.2056408Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:27.2057149Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:27.2057987Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.2064509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.2065450Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.2066400Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.2067400Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.2068374Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.2069474Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.2071165Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.2072186Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.2073354Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.2074653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.2076359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.2078316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.2079473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.2081082Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.2082530Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.2083580Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.2084594Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.2124880Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.2165403Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.2168268Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.2211781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.2213295Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.2214718Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.2216203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.2217615Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.2218995Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.2220388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.2221765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.2223118Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.2224399Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.2225648Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.2226866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.2228085Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.2229274Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.2230487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.2287591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.2347740Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.2349083Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.2350073Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.2350997Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.2351907Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.2352836Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:27.2353862Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2355017Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2356122Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:27.2357175Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:27.2358259Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2359337Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:27.2360414Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2361546Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:27.2362608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:27.2380246Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:27.2524575Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:27.2526566Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:27.2528750Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2530656Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2531821Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:27.2532978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:27.2534107Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2535251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:27.2536392Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2537519Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:27.2538634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:27.2553179Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:27.2637065Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:27.2638310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.2639401Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.2640694Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.2641630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.2642698Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.2643607Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:27.2644652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:27.2645756Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:27.2646700Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:27.2766970Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:27.2830455Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:27.2831307Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:27.2832013Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:27.2832829Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.2839058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.2839984Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.2840904Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.2841884Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.2842844Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.2843929Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.2845324Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.2846314Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.2847298Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.2849794Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.2851240Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.2853564Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.2854700Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.2855967Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.2857270Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.2858308Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.2859301Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.2899254Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.2939008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.2940532Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.2985808Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.2987218Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.2988834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.2990410Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.2991804Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.2993171Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.2994544Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.2995926Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.2997270Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.2998792Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3000028Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3001239Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3002450Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3003629Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3004820Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3060861Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.3121327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.3122519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.3123514Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.3124369Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.3125266Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.3126206Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:27.3127260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:27.3128432Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:27.3129577Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:27.3130695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:27.3131818Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:27.3133007Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:27.3134110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:27.3135210Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:27.3136305Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:27.3153297Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:27.3218139Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:27.3219320Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3220508Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3221479Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3222372Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3223332Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3224265Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:27.3226683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:27.3227467Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:27.3228644Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:27.3278141Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:27.3278928Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:27.3279686Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:27.3280619Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3287060Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3288063Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3289055Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3290111Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3291195Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3292464Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3293892Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3294980Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3296029Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3297584Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3299049Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3300925Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3302129Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3303640Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3305079Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3306179Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3307236Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3346801Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3386352Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3388064Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3433533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3435171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3436642Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3438160Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3439621Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3441124Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3442570Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3444009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3445416Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3446753Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3448043Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3449317Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3450593Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3451849Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3453203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3509350Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.3569398Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.3570645Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.3571685Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.3572719Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.3573688Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.3574703Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:27.3575836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.3577089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.3578300Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:27.3579441Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:27.3580623Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.3581990Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:27.3583186Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.3584486Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:27.3585665Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:27.3586911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:27.3711111Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:27.3712292Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:27.3713485Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.3714799Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.3716070Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:27.3717259Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:27.3718502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.3719737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:27.3721025Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.3722258Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:27.3723477Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:27.3740576Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:27.3795958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:27.3797239Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3798585Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3799624Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3800590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3801556Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3802521Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:27.3804993Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:27.3805862Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:27.3806855Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:27.3910669Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:27.3953726Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:27.3954431Z creating: build/lib/ 2024-08-20T22:20:27.4043178Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:27.4499026Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:27.4508745Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:27.4517131Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:27.4525268Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:27.4526007Z inflating: build/lib/libclog.a 2024-08-20T22:20:27.4544233Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:27.4546979Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:27.4612120Z inflating: build/lib/libgtest.a 2024-08-20T22:20:27.4686775Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:27.4750664Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:27.4758286Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:27.4786161Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:27.4913374Z inflating: build/lib/libgloo.a 2024-08-20T22:20:27.4934280Z inflating: build/lib/libfmt.a 2024-08-20T22:20:27.5028990Z inflating: build/lib/libc10.so 2024-08-20T22:20:27.5031078Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:27.5032644Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:27.5052144Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:27.5556166Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:27.5574518Z inflating: build/lib/libgmock.a 2024-08-20T22:20:27.5575210Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:27.5576199Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:27.5966036Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:28.6151549Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:28.6726791Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:28.6783550Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:28.6784254Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:28.8044137Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:28.8300033Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:28.8806409Z inflating: build/lib/libkineto.a 2024-08-20T22:20:28.8990139Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:28.9033381Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:28.9726931Z inflating: build/lib/libonnx.a 2024-08-20T22:20:31.4198497Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:31.4203732Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:31.4207687Z inflating: build/lib/libshm.so 2024-08-20T22:20:33.4838365Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:33.4840636Z inflating: build/lib/libtorch.so 2024-08-20T22:20:33.4842549Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:34.3006534Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:34.5002837Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:34.5075416Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:34.5096183Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:34.5122672Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:34.5148486Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:34.5183397Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:34.5183879Z creating: build/bin/ 2024-08-20T22:20:34.5234616Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:34.5286868Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:34.5338663Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:34.5397772Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:34.5451785Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:34.5500946Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:34.5552672Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:34.5607953Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:34.5664542Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:34.5721128Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:34.5792562Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:34.5846712Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:34.5895667Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:34.5946173Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:34.5997169Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:34.6054070Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:34.6109905Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:34.6160654Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:34.6216614Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:34.6268256Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:34.6320165Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:34.6370023Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:34.6422126Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:34.6478809Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:34.6529473Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:34.6585968Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:34.6642238Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:34.6695536Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:34.6746631Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:34.6797697Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:34.6963714Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:34.7015517Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:34.7069270Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:34.7127308Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:34.7190043Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:34.7265678Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:34.7418013Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:34.7472515Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:34.7525086Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:34.7576843Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:34.7627726Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:34.7686951Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:34.7736075Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:34.7792706Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:34.8241311Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:34.8689182Z inflating: build/bin/protoc 2024-08-20T22:20:34.8742936Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:34.8796614Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:34.8849592Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:34.8901905Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:34.8955954Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:34.9008430Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:34.9057877Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:34.9111853Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:34.9444423Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:34.9791034Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:35.0152678Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:35.0205771Z inflating: build/bin/BackoffTest 2024-08-20T22:20:35.0260022Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:35.0316012Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:35.0370003Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:35.0383981Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:35.0438996Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:35.0443198Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:35.0446240Z inflating: build/bin/example_allreduce 2024-08-20T22:20:35.0502409Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:35.0571831Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:35.0574771Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:35.0642676Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:35.0652065Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:35.1000710Z inflating: build/bin/test_lazy 2024-08-20T22:20:35.2186796Z inflating: build/bin/test_api 2024-08-20T22:20:35.2260784Z inflating: build/bin/Dict_test 2024-08-20T22:20:35.2314274Z inflating: build/bin/Dimname_test 2024-08-20T22:20:35.2379419Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:35.2436988Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:35.2497692Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:35.2556563Z inflating: build/bin/atest 2024-08-20T22:20:35.2620569Z inflating: build/bin/basic 2024-08-20T22:20:35.2676109Z inflating: build/bin/broadcast_test 2024-08-20T22:20:35.2727541Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:35.2786852Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:35.2840440Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:35.2934501Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:35.2985018Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:35.3036542Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:35.3094857Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:35.3150835Z inflating: build/bin/half_test 2024-08-20T22:20:35.3247570Z inflating: build/bin/ivalue_test 2024-08-20T22:20:35.3297996Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:35.3353401Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:35.3407692Z inflating: build/bin/memory_format_test 2024-08-20T22:20:35.3461468Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:35.3515312Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:35.3572768Z inflating: build/bin/native_test 2024-08-20T22:20:35.3623684Z inflating: build/bin/operator_name_test 2024-08-20T22:20:35.3676182Z inflating: build/bin/operators_test 2024-08-20T22:20:35.3728073Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:35.3796429Z inflating: build/bin/pow_test 2024-08-20T22:20:35.3853776Z inflating: build/bin/quantized_test 2024-08-20T22:20:35.3904583Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:35.3956342Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:35.4013530Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:35.4072771Z inflating: build/bin/scalar_test 2024-08-20T22:20:35.4125923Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:35.4178575Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:35.4258515Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:35.4313302Z inflating: build/bin/test_parallel 2024-08-20T22:20:35.4316550Z inflating: build/bin/thread_init_test 2024-08-20T22:20:35.4372803Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:35.4434085Z inflating: build/bin/type_test 2024-08-20T22:20:35.4486579Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:35.4488021Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:35.4558092Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:35.4610514Z inflating: build/bin/weakref_test 2024-08-20T22:20:35.4662413Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:35.4714972Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:35.4775510Z inflating: build/bin/IListRef_test 2024-08-20T22:20:35.4883053Z inflating: build/bin/List_test 2024-08-20T22:20:35.4950622Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:35.5073564Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:35.5171295Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:35.5300910Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:35.5405259Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:35.5466835Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:35.5564561Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:35.5616563Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:35.5672403Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:35.5722096Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:35.6027448Z inflating: build/bin/op_registration_test 2024-08-20T22:20:35.6091800Z inflating: build/bin/inline_container_test 2024-08-20T22:20:35.6145680Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:35.6199017Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:35.6257720Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:35.6313152Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:35.6383372Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:35.6443327Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:35.6493716Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:35.6551302Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:35.6602977Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:35.6668300Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:35.6725785Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:35.6775923Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:35.6827713Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:35.6877785Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:35.6929780Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:35.6982981Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:35.7034194Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:35.7094522Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:35.7144797Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:35.7197190Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:35.7212252Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:35.7278186Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:35.7336539Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:35.7400990Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:35.7463691Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:35.8295159Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:35.8878374Z inflating: build/bin/test_jit 2024-08-20T22:20:35.8878818Z creating: .additional_ci_files/ 2024-08-20T22:20:35.8939835Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:35.9181089Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:35.9218874Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:35.9219274Z rm artifacts.zip 2024-08-20T22:20:35.9228354Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:35.9228858Z env: 2024-08-20T22:20:35.9229141Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:35.9229593Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:35.9230083Z ##[endgroup] 2024-08-20T22:20:36.0340651Z ##[group]Run df -H 2024-08-20T22:20:36.0340978Z df -H 2024-08-20T22:20:36.0349878Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.0350375Z env: 2024-08-20T22:20:36.0350650Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.0351102Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.0351584Z ##[endgroup] 2024-08-20T22:20:36.0401964Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:36.0402619Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:36.0403987Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:36.0404491Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:36.0405016Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:36.0405759Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:36.0406253Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:36.0406896Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:36.0447304Z Prepare all required actions 2024-08-20T22:20:36.0447778Z Getting action download info 2024-08-20T22:20:36.2104264Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:36.2104725Z with: 2024-08-20T22:20:36.2104985Z env: 2024-08-20T22:20:36.2105261Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2105715Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2106190Z ##[endgroup] 2024-08-20T22:20:36.2139718Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:36.2140163Z with: 2024-08-20T22:20:36.2140429Z name: td_results 2024-08-20T22:20:36.2140748Z s3-bucket: gha-artifacts 2024-08-20T22:20:36.2141099Z region: us-east-1 2024-08-20T22:20:36.2141396Z env: 2024-08-20T22:20:36.2141666Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2142121Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2142603Z ##[endgroup] 2024-08-20T22:20:36.6857581Z (node:52519) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:36.6858283Z 2024-08-20T22:20:36.6858540Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:36.6859262Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:36.6860111Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:36.7454343Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:36.7460847Z Artifact download has finished successfully 2024-08-20T22:20:36.7782949Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:36.7783438Z mkdir -p .additional_ci_files 2024-08-20T22:20:36.7783996Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:36.7793005Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.7793510Z env: 2024-08-20T22:20:36.7793794Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7794281Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7794762Z ##[endgroup] 2024-08-20T22:20:36.7854211Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:36.7867374Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:36.7902376Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:36.7902860Z .github/scripts/parse_ref.py 2024-08-20T22:20:36.7911910Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:36.7912265Z env: 2024-08-20T22:20:36.7912543Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7912996Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7913476Z ##[endgroup] 2024-08-20T22:20:36.8197254Z Prepare all required actions 2024-08-20T22:20:36.8264160Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:36.8264614Z with: 2024-08-20T22:20:36.8265154Z github-token: *** 2024-08-20T22:20:36.8265463Z env: 2024-08-20T22:20:36.8265739Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8266210Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8266694Z ##[endgroup] 2024-08-20T22:20:36.8283308Z ##[group]Run set -eux 2024-08-20T22:20:36.8283641Z set -eux 2024-08-20T22:20:36.8284219Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:36.8293265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.8293780Z env: 2024-08-20T22:20:36.8294085Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8294540Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8295174Z GITHUB_TOKEN: *** 2024-08-20T22:20:36.8295480Z ##[endgroup] 2024-08-20T22:20:36.8323866Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-0d6323d38c97c2036 2024-08-20T22:20:38.3421763Z setting job-id=29026530597 2024-08-20T22:20:38.3423469Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.3641786Z Prepare all required actions 2024-08-20T22:20:38.3642245Z Getting action download info 2024-08-20T22:20:38.4706783Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:38.4707237Z with: 2024-08-20T22:20:38.4707698Z github-token: *** 2024-08-20T22:20:38.4716416Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:38.4725859Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.4726615Z env: 2024-08-20T22:20:38.4726897Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.4727356Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.4727836Z ##[endgroup] 2024-08-20T22:20:38.4770036Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:38.4770577Z with: 2024-08-20T22:20:38.4770841Z shell: bash 2024-08-20T22:20:38.4771145Z timeout_minutes: 10 2024-08-20T22:20:38.4771468Z max_attempts: 5 2024-08-20T22:20:38.4771781Z retry_wait_seconds: 30 2024-08-20T22:20:38.4772975Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.4774118Z polling_interval_seconds: 1 2024-08-20T22:20:38.4774483Z warning_on_retry: true 2024-08-20T22:20:38.4774820Z continue_on_error: false 2024-08-20T22:20:38.4775155Z env: 2024-08-20T22:20:38.4775422Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.4775859Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.4776492Z GITHUB_TOKEN: *** 2024-08-20T22:20:38.4776802Z ##[endgroup] 2024-08-20T22:20:38.5526875Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.7891182Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:38.9356195Z Collecting requests==2.27.1 2024-08-20T22:20:38.9708018Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:39.1642191Z Collecting pyyaml==6.0.1 2024-08-20T22:20:39.1683848Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:39.2444960Z Collecting certifi>=2017.4.17 2024-08-20T22:20:39.2485937Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:39.2540328Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:39.2548691Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:39.5514586Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:39.5557711Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:39.6346283Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:39.7561518Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:40.5315337Z Command completed after 1 attempt(s). 2024-08-20T22:20:40.5375955Z ##[group]Run set -x 2024-08-20T22:20:40.5376286Z set -x 2024-08-20T22:20:40.5376580Z  2024-08-20T22:20:40.5377120Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.5377784Z # in runner workspace 2024-08-20T22:20:40.5378307Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:40.5387307Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.5387803Z env: 2024-08-20T22:20:40.5388074Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.5388527Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.5389010Z ##[endgroup] 2024-08-20T22:20:40.5420307Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:40.5658802Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.5659359Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.5659836Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:40.5660241Z  2024-08-20T22:20:40.5660804Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.5661493Z # in runner workspace 2024-08-20T22:20:40.5662081Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:40.5662723Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:40.5663191Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:40.5672319Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:40.5681658Z  --selected-test-configs "" \ 2024-08-20T22:20:40.5682136Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:40.5682577Z  --tag "${TAG}" \ 2024-08-20T22:20:40.5682982Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:40.5683433Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:40.5683870Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:40.5692867Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.5693380Z env: 2024-08-20T22:20:40.5693669Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.5694140Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.5694887Z GITHUB_TOKEN: *** 2024-08-20T22:20:40.5695603Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.5696381Z PR_NUMBER: 2024-08-20T22:20:40.5696702Z TAG: ciflow/inductor/133712 2024-08-20T22:20:40.5697087Z EVENT_NAME: push 2024-08-20T22:20:40.5697709Z SCHEDULE: 2024-08-20T22:20:40.5698011Z HEAD_BRANCH: 2024-08-20T22:20:40.5698331Z ##[endgroup] 2024-08-20T22:20:40.5727542Z Workflow: inductor 2024-08-20T22:20:40.5728561Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.8079237Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:40.9428760Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:40.9429216Z echo "Filtered matrix:" 2024-08-20T22:20:40.9438211Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:40.9447419Z  2024-08-20T22:20:40.9447696Z echo 2024-08-20T22:20:40.9448072Z echo "Is the current job unstable? False" 2024-08-20T22:20:40.9448520Z  2024-08-20T22:20:40.9448791Z echo 2024-08-20T22:20:40.9449153Z echo "Is keep-going label set? False" 2024-08-20T22:20:40.9449590Z  2024-08-20T22:20:40.9449862Z echo 2024-08-20T22:20:40.9450183Z echo "Renabled issues? " 2024-08-20T22:20:40.9459219Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.9459713Z env: 2024-08-20T22:20:40.9459991Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.9460448Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.9460931Z ##[endgroup] 2024-08-20T22:20:40.9490633Z Filtered matrix: 2024-08-20T22:20:40.9502822Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:40.9511661Z 2024-08-20T22:20:40.9511810Z Is the current job unstable? False 2024-08-20T22:20:40.9512094Z 2024-08-20T22:20:40.9512383Z Is keep-going label set? False 2024-08-20T22:20:40.9512645Z 2024-08-20T22:20:40.9512768Z Renabled issues? 2024-08-20T22:20:40.9564662Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:40.9565377Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:40.9574429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.9574918Z env: 2024-08-20T22:20:40.9575203Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.9575661Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.9576323Z JOB_TIMEOUT: 240 2024-08-20T22:20:40.9576632Z ##[endgroup] 2024-08-20T22:20:40.9658175Z ##[group]Run set -x 2024-08-20T22:20:40.9658566Z set -x 2024-08-20T22:20:40.9658859Z  2024-08-20T22:20:40.9659212Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:40.9659743Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:40.9660299Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:40.9660800Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:40.9661209Z else 2024-08-20T22:20:40.9661540Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:40.9661955Z fi 2024-08-20T22:20:40.9662229Z  2024-08-20T22:20:40.9662690Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:40.9663432Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:40.9664116Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:40.9664695Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:40.9665147Z container_name=$(docker run \ 2024-08-20T22:20:40.9665562Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:40.9665932Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:40.9666330Z  -e PR_NUMBER \ 2024-08-20T22:20:40.9666683Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:40.9667074Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:40.9667474Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:40.9667853Z  -e GITHUB_JOB \ 2024-08-20T22:20:40.9668209Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:40.9668592Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:40.9668993Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:40.9669384Z  -e JOB_ID \ 2024-08-20T22:20:40.9669718Z  -e JOB_NAME \ 2024-08-20T22:20:40.9670063Z  -e BASE_SHA \ 2024-08-20T22:20:40.9670410Z  -e BRANCH \ 2024-08-20T22:20:40.9670737Z  -e SHA1 \ 2024-08-20T22:20:40.9671080Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:40.9671483Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:40.9671861Z  -e SHARD_NUMBER \ 2024-08-20T22:20:40.9672219Z  -e TEST_CONFIG \ 2024-08-20T22:20:40.9672590Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:40.9672980Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:40.9673390Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:40.9673803Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:40.9674201Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:40.9674590Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:40.9674961Z  -e NO_TD \ 2024-08-20T22:20:40.9675321Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:40.9675724Z  -e PR_LABELS \ 2024-08-20T22:20:40.9676128Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:40.9676573Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:40.9676970Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:40.9677376Z  -e XLA_CUDA \ 2024-08-20T22:20:40.9677780Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:40.9678280Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:40.9678798Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:40.9679305Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:40.9679768Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:40.9680211Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:40.9680646Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:40.9681119Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:40.9681671Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:40.9682125Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:40.9682515Z  --ipc=host \ 2024-08-20T22:20:40.9682873Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:40.9683428Z  --tty \ 2024-08-20T22:20:40.9683733Z  --detach \ 2024-08-20T22:20:40.9684200Z  --name="${container_name}" \ 2024-08-20T22:20:40.9684614Z  --user jenkins \ 2024-08-20T22:20:40.9685141Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:40.9685697Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:40.9686127Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:40.9686476Z ) 2024-08-20T22:20:40.9686883Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:40.9687787Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:40.9688742Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:40.9689642Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:40.9698719Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:40.9699074Z env: 2024-08-20T22:20:40.9699357Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.9699812Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.9700410Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:40.9700905Z PR_NUMBER: 2024-08-20T22:20:40.9701226Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:40.9701639Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:40.9701989Z GITHUB_JOB: test 2024-08-20T22:20:40.9702298Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:40.9702663Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:40.9703013Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:40.9703347Z JOB_ID: 29026530597 2024-08-20T22:20:40.9704035Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.9704789Z BRANCH: 2024-08-20T22:20:40.9705129Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:40.9705640Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:40.9706122Z TEST_CONFIG: aot_inductor_torchbench 2024-08-20T22:20:40.9706531Z SHARD_NUMBER: 1 2024-08-20T22:20:40.9706838Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:40.9707160Z REENABLED_ISSUES: 2024-08-20T22:20:40.9707490Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:40.9707873Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:40.9708228Z TEST_SHOWLOCALS: False 2024-08-20T22:20:40.9708565Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:40.9708896Z NO_TD: False 2024-08-20T22:20:40.9709198Z TD_DISTRIBUTED: False 2024-08-20T22:20:40.9709612Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:40.9710091Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:40.9710466Z SHM_SIZE: 2g 2024-08-20T22:20:40.9711541Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:40.9712690Z XLA_CUDA: 2024-08-20T22:20:40.9713163Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:40.9713779Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:40.9714211Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:40.9714615Z DASHBOARD_TAG: 2024-08-20T22:20:40.9715125Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:40.9715498Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:40.9715868Z ##[endgroup] 2024-08-20T22:20:40.9743755Z + [[ aot_inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:40.9744814Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:40.9745352Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:40.9752544Z +++ nproc --ignore=2 2024-08-20T22:20:40.9806823Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:52.3609243Z + container_name=7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T22:20:52.3610667Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:52.3613945Z + docker exec -i 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:52.5282719Z + echo DOCKER_CONTAINER_ID=7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T22:20:52.5286708Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:52.5289640Z + docker exec -t 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:52.9307024Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:53.2599279Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:53.2601791Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:53.2604452Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:53.2607851Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:53.2610372Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:53.2615674Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:53.2631702Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:53.2644678Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:53.2661977Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:53.3058619Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:20:53.8930605Z Installing collected packages: torch 2024-08-20T22:21:03.6650868Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:03.6653170Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:03.6654191Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:03.7568235Z + export TERM=vt100 2024-08-20T22:21:03.7568572Z + TERM=vt100 2024-08-20T22:21:03.7571441Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:03.7582640Z + source .ci/pytorch/common.sh 2024-08-20T22:21:03.7585645Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:03.7594725Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:03.7596859Z +++ declare -f -t trap_add 2024-08-20T22:21:03.7602387Z ++ set -ex 2024-08-20T22:21:03.7602873Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:03.7603447Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:03.7604228Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:03.7606960Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:03.7626920Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:03.7627368Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:03.7627797Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:03.7628258Z + shift 2024-08-20T22:21:03.7628542Z + for trap_add_name in "$@" 2024-08-20T22:21:03.7635511Z +++ trap -p EXIT 2024-08-20T22:21:03.7639096Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:03.7639467Z +++ extract_trap_cmd 2024-08-20T22:21:03.7639916Z +++ printf '%s\n' '' 2024-08-20T22:21:03.7640463Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:03.7643012Z + trap -- ' 2024-08-20T22:21:03.7643455Z cleanup_workspace' EXIT 2024-08-20T22:21:03.7644110Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:04.4745360Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:04.4767931Z + echo 'Environment variables:' 2024-08-20T22:21:04.4768458Z Environment variables: 2024-08-20T22:21:04.4768883Z + env 2024-08-20T22:21:04.4779012Z INSTALLED_DB=yes 2024-08-20T22:21:04.4780590Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.4781016Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.4781403Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.4782015Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.4782584Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.4783088Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.4783712Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.4784542Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.4785035Z HOSTNAME=7f35680bf802 2024-08-20T22:21:04.4785880Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.4786838Z GITHUB_ACTION=__self 2024-08-20T22:21:04.4787287Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.4791503Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.4795363Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.4795872Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.4796227Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.4796622Z TEST_CONFIG=aot_inductor_torchbench 2024-08-20T22:21:04.4797040Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.4799663Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.4800151Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.4800829Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.4801382Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.4801778Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.4802224Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.4802711Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.4803172Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.4803491Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.4803868Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.4804256Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.4804689Z XLA_CUDA= 2024-08-20T22:21:04.4805183Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.4807342Z *** 2024-08-20T22:21:04.4807641Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.4808042Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.4808397Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.4808731Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.4809254Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.4809793Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.4810266Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.4810780Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.4811262Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.4812039Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.4812869Z UCC_HOME=/usr 2024-08-20T22:21:04.4813217Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.4813613Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.4813946Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.4814416Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.4814907Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.4815371Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.4815741Z SHARD_NUMBER=1 2024-08-20T22:21:04.4816047Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.4816394Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.4816760Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.4817195Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.4817676Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.4818163Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.4818522Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.4818949Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.4819382Z NUM_TEST_SHARDS=2 2024-08-20T22:21:04.4819681Z UCX_HOME=/usr 2024-08-20T22:21:04.4820190Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.4821288Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.4822584Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.4823832Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.4824937Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.4825589Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.4825916Z DASHBOARD_TAG= 2024-08-20T22:21:04.4826223Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.4826677Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.4827219Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.4828205Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.4829105Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.4829506Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.4829881Z PR_NUMBER= 2024-08-20T22:21:04.4830164Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.4830494Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.4830934Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.4831389Z TERM=vt100 2024-08-20T22:21:04.4831724Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.4832247Z INSTALLED_VISION=yes 2024-08-20T22:21:04.4832561Z BRANCH= 2024-08-20T22:21:04.4832931Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.4833330Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.4833744Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.4834504Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.4835253Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.4835730Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.4836197Z REENABLED_ISSUES= 2024-08-20T22:21:04.4836495Z SHLVL=1 2024-08-20T22:21:04.4836755Z MAX_JOBS=14 2024-08-20T22:21:04.4837064Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.4837417Z NVARCH=x86_64 2024-08-20T22:21:04.4837714Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.4838170Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.4838691Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.4839177Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.4839848Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.4840418Z GITHUB_JOB=test 2024-08-20T22:21:04.4840813Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.4841395Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.4841926Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.4842248Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.4842624Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.4843056Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.4843474Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.4843829Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.4844173Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.4844522Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.4845547Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.4846589Z GITHUB_BASE_REF= 2024-08-20T22:21:04.4846927Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.4847296Z CI=true 2024-08-20T22:21:04.4847609Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.4848018Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.4848387Z JOB_ID=29026530597 2024-08-20T22:21:04.4848698Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.4849019Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.4849324Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.4849762Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.4850221Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.4850545Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.4850904Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.4851837Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.4852753Z NO_TD=False 2024-08-20T22:21:04.4853048Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.4853408Z _=/usr/bin/env 2024-08-20T22:21:04.4853894Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:04.5010056Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:04.5011228Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:04.5012262Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:04.5013147Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:04.5013749Z + BUILD_DIR=build 2024-08-20T22:21:04.5014068Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:04.5014455Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:04.5014811Z + SHARD_NUMBER=1 2024-08-20T22:21:04.5015103Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:04.5015421Z + export VALGRIND=ON 2024-08-20T22:21:04.5015737Z + VALGRIND=ON 2024-08-20T22:21:04.5016331Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:04.5016835Z + [[ 0 == \1 ]] 2024-08-20T22:21:04.5017130Z + [[ False == \1 ]] 2024-08-20T22:21:04.5017789Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:04.5018561Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:04.5028633Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:04.5029408Z + [[ -n '' ]] 2024-08-20T22:21:04.5029770Z + echo 'Environment variables' 2024-08-20T22:21:04.5030145Z Environment variables 2024-08-20T22:21:04.5030454Z + env 2024-08-20T22:21:04.5038286Z INSTALLED_DB=yes 2024-08-20T22:21:04.5038942Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.5039474Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.5040038Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.5040815Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.5041576Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.5042137Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.5042768Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.5043575Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.5044107Z HOSTNAME=7f35680bf802 2024-08-20T22:21:04.5044956Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.5045801Z GITHUB_ACTION=__self 2024-08-20T22:21:04.5046181Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.5050690Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.5054779Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.5055293Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.5055647Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.5056006Z TEST_CONFIG=aot_inductor_torchbench 2024-08-20T22:21:04.5056415Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.5056889Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.5057361Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.5057793Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.5058199Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.5058591Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.5059038Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.5059523Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.5059981Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.5060307Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.5060689Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.5061077Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5061556Z XLA_CUDA= 2024-08-20T22:21:04.5062022Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.5062410Z *** 2024-08-20T22:21:04.5062685Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.5063087Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.5063449Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.5063779Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.5064268Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.5064802Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.5065270Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5065772Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.5066252Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5067043Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.5067901Z UCC_HOME=/usr 2024-08-20T22:21:04.5068234Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.5068723Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.5069061Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.5069528Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.5070015Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.5070479Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.5070854Z SHARD_NUMBER=1 2024-08-20T22:21:04.5071162Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.5071507Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.5071871Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.5072309Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.5072777Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.5073261Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.5073627Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.5074050Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.5074479Z NUM_TEST_SHARDS=2 2024-08-20T22:21:04.5074790Z UCX_HOME=/usr 2024-08-20T22:21:04.5075305Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.5076379Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.5077648Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.5078902Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.5080002Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.5080653Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.5080975Z DASHBOARD_TAG= 2024-08-20T22:21:04.5081288Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.5081748Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.5082286Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.5083267Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.5084182Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.5084588Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.5084960Z PR_NUMBER= 2024-08-20T22:21:04.5085240Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.5085555Z VALGRIND=ON 2024-08-20T22:21:04.5085853Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.5086289Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.5086741Z TERM=vt100 2024-08-20T22:21:04.5087084Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.5087493Z INSTALLED_VISION=yes 2024-08-20T22:21:04.5087797Z BRANCH= 2024-08-20T22:21:04.5088081Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.5088481Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.5088908Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.5089652Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.5090400Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.5090888Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.5091349Z REENABLED_ISSUES= 2024-08-20T22:21:04.5091640Z SHLVL=1 2024-08-20T22:21:04.5091898Z MAX_JOBS=14 2024-08-20T22:21:04.5092343Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.5092720Z NVARCH=x86_64 2024-08-20T22:21:04.5093023Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.5093480Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5094009Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.5094493Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.5095159Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.5095721Z GITHUB_JOB=test 2024-08-20T22:21:04.5096122Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.5096663Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.5097171Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.5098004Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.5098417Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.5099009Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.5099440Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.5099800Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.5100139Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.5100490Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.5101516Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.5102567Z GITHUB_BASE_REF= 2024-08-20T22:21:04.5102884Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.5103252Z CI=true 2024-08-20T22:21:04.5103567Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.5103980Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.5104343Z JOB_ID=29026530597 2024-08-20T22:21:04.5104655Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.5104981Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.5105287Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.5105725Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.5106191Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.5106524Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.5106872Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.5107770Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6c7434be-f469-48d6-9714-1b78e42f64be 2024-08-20T22:21:04.5108569Z NO_TD=False 2024-08-20T22:21:04.5108871Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.5109222Z _=/usr/bin/env 2024-08-20T22:21:04.5109561Z + echo 'Testing pytorch' 2024-08-20T22:21:04.5109900Z Testing pytorch 2024-08-20T22:21:04.5110213Z + export LANG=C.UTF-8 2024-08-20T22:21:04.5110552Z + LANG=C.UTF-8 2024-08-20T22:21:04.5110854Z + PR_NUMBER= 2024-08-20T22:21:04.5111208Z + [[ aot_inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:04.5111734Z + [[ aot_inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:04.5112252Z + [[ aot_inductor_torchbench == \s\l\o\w ]] 2024-08-20T22:21:04.5112906Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:04.5113603Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:04.5114139Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.5114609Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.5115069Z + [[ aot_inductor_torchbench == *crossref* ]] 2024-08-20T22:21:04.5115656Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:04.5116280Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:04.5116927Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:04.5117491Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:04.5118025Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:05.3628445Z Collecting ninja==1.10.2 2024-08-20T22:21:05.3775366Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:05.4273138Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:06.0300480Z Installing collected packages: ninja 2024-08-20T22:21:06.0377727Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:06.0378989Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:06.0876142Z Successfully installed ninja-1.10.2 2024-08-20T22:21:06.1692742Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.1695053Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.1696475Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:06.1697198Z + install_tlparse 2024-08-20T22:21:06.1697951Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:06.1698716Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:06.5917601Z Collecting tlparse==0.3.25 2024-08-20T22:21:06.6072296Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:06.6562363Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:07.2895281Z Installing collected packages: tlparse 2024-08-20T22:21:07.3650520Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:07.4458373Z ++ python -m site --user-base 2024-08-20T22:21:07.4704213Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:07.4705896Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:07.4706581Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:07.4707237Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:07.4708284Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:07.4709728Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:07.4710607Z + cd test 2024-08-20T22:21:07.4711140Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:09.1082018Z + [[ aot_inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:09.1082665Z + [[ aot_inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:09.1085846Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:09.1086419Z + [[ aot_inductor_torchbench == *pr_time_benchmarks* ]] 2024-08-20T22:21:09.1087031Z + [[ aot_inductor_torchbench == *dynamo_eager* ]] 2024-08-20T22:21:09.1087589Z + [[ aot_inductor_torchbench == *aot_eager* ]] 2024-08-20T22:21:09.1088115Z + [[ aot_inductor_torchbench == *aot_inductor* ]] 2024-08-20T22:21:09.1088923Z + DYNAMO_BENCHMARK_FLAGS+=(--export-aot-inductor) 2024-08-20T22:21:09.1089500Z + [[ aot_inductor_torchbench == *dynamic* ]] 2024-08-20T22:21:09.1089977Z + [[ aot_inductor_torchbench == *cpu* ]] 2024-08-20T22:21:09.1090559Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:09.1118885Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:09.1119634Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:09.1121051Z + cd test 2024-08-20T22:21:09.1121563Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:10.5805391Z PyTorch built with: 2024-08-20T22:21:10.5806132Z - GCC 9.4 2024-08-20T22:21:10.5806568Z - C++ Version: 201703 2024-08-20T22:21:10.5807646Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:10.5808896Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:10.5809577Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:10.5810095Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:10.5810570Z - NNPACK is enabled 2024-08-20T22:21:10.5810959Z - CPU capability usage: AVX2 2024-08-20T22:21:10.5811370Z - CUDA Runtime 12.1 2024-08-20T22:21:10.5811892Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:10.5812603Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:10.5813058Z - Magma 2.6.1 2024-08-20T22:21:10.5821538Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:10.5828032Z 2024-08-20T22:21:10.9184604Z + cd test 2024-08-20T22:21:10.9185502Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:12.2540665Z ATen/Parallel: 2024-08-20T22:21:12.2541617Z at::get_num_threads() : 8 2024-08-20T22:21:12.2542594Z at::get_num_interop_threads() : 16 2024-08-20T22:21:12.2543688Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:12.2544372Z omp_get_max_threads() : 8 2024-08-20T22:21:12.2545430Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:12.2546223Z mkl_get_max_threads() : 8 2024-08-20T22:21:12.2546850Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:12.2547472Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:12.2547889Z Environment variables: 2024-08-20T22:21:12.2548236Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:12.2548605Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:12.2548977Z ATen parallel backend: OpenMP 2024-08-20T22:21:12.2549227Z 2024-08-20T22:21:12.5397094Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:12.5398123Z + [[ aot_inductor_torchbench == *backward* ]] 2024-08-20T22:21:12.5398743Z + [[ aot_inductor_torchbench == *xla* ]] 2024-08-20T22:21:12.5399240Z + [[ aot_inductor_torchbench == *executorch* ]] 2024-08-20T22:21:12.5399757Z + [[ aot_inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:12.5400478Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:12.5401177Z + [[ aot_inductor_torchbench == distributed ]] 2024-08-20T22:21:12.5401776Z + [[ aot_inductor_torchbench == *inductor_distributed* ]] 2024-08-20T22:21:12.5402408Z + [[ aot_inductor_torchbench == *inductor-halide* ]] 2024-08-20T22:21:12.5403047Z + [[ aot_inductor_torchbench == *inductor-micro-benchmark* ]] 2024-08-20T22:21:12.5403604Z + [[ aot_inductor_torchbench == *huggingface* ]] 2024-08-20T22:21:12.5404132Z + [[ aot_inductor_torchbench == *timm* ]] 2024-08-20T22:21:12.5404601Z + [[ aot_inductor_torchbench == *torchbench* ]] 2024-08-20T22:21:12.5405073Z + [[ aot_inductor_torchbench == *cpu* ]] 2024-08-20T22:21:12.5405493Z + install_torchaudio cuda 2024-08-20T22:21:12.5405838Z + local commit 2024-08-20T22:21:12.5406157Z ++ get_pinned_commit audio 2024-08-20T22:21:12.5406536Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T22:21:12.5419426Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:12.5420023Z + [[ cuda == \c\u\d\a ]] 2024-08-20T22:21:12.5420469Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:21:12.5421385Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:12.5422816Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:12.8765530Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:12.8768361Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-3nd3sljw 2024-08-20T22:21:12.8799743Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-3nd3sljw 2024-08-20T22:21:13.6760297Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T22:21:13.6787551Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:14.0805504Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:14.0806623Z Running command git submodule update --init --recursive -q 2024-08-20T22:21:16.5778100Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:21:16.5807452Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:16.5836954Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T22:21:16.5841734Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T22:21:16.5845547Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T22:21:16.5847360Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T22:21:16.5849776Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T22:21:16.5855464Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1) 2024-08-20T22:21:16.5870272Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T22:21:16.6349469Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T22:21:16.6412986Z Building wheels for collected packages: torchaudio 2024-08-20T22:23:50.5742533Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:23:50.5780697Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310397 sha256=f7b328e91e47bb653ef9be0e13b16d0083ca6e177fecf0608c879173dc383bc1 2024-08-20T22:23:50.5783148Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95 2024-08-20T22:23:50.5814913Z Successfully built torchaudio 2024-08-20T22:23:51.0755447Z Installing collected packages: torchaudio 2024-08-20T22:23:51.2938205Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T22:23:51.4747390Z + install_torchtext 2024-08-20T22:23:51.4747747Z + local data_commit 2024-08-20T22:23:51.4748072Z + local text_commit 2024-08-20T22:23:51.4749604Z ++ get_pinned_commit data 2024-08-20T22:23:51.4750069Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T22:23:51.4774795Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.4779486Z ++ get_pinned_commit text 2024-08-20T22:23:51.4779946Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T22:23:51.4798311Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:23:51.4800088Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.4801525Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.8179206Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.8183548Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-18ytlcwl 2024-08-20T22:23:51.8220208Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-18ytlcwl 2024-08-20T22:23:52.2558464Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T22:23:52.2586546Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.5174115Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.6671126Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.6672403Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:14.4252002Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:14.4291201Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T22:25:14.4293483Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T22:25:14.4298211Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:14.4328258Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T22:25:14.4331975Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T22:25:14.4335855Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T22:25:14.4338734Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T22:25:14.4341827Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T22:25:14.4346822Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1) 2024-08-20T22:25:14.4361261Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T22:25:14.4421065Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T22:25:14.4424655Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T22:25:14.4429610Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T22:25:14.4930704Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T22:25:14.4995323Z Building wheels for collected packages: torchdata 2024-08-20T22:25:16.4607173Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T22:25:16.4615875Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=80d6f64337b79b7f468121e687c4678b834c06079eebe723f49bfbe199b93c14 2024-08-20T22:25:16.4617429Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-08-20T22:25:16.4645369Z Successfully built torchdata 2024-08-20T22:25:16.9467182Z Installing collected packages: torchdata 2024-08-20T22:25:17.0475505Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T22:25:21.9531180Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:21.9532765Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:22.2895365Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:22.2899880Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-fk4ujtyk 2024-08-20T22:25:22.2931726Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-fk4ujtyk 2024-08-20T22:25:23.0990221Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T22:25:23.1016947Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:23.5903556Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:23.7583561Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:23.7584648Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:31.3539824Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:31.3576279Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T22:25:31.3578767Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T22:25:31.3582454Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:31.3585535Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4) 2024-08-20T22:25:31.3588807Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T22:25:31.3615846Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T22:25:31.3619631Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T22:25:31.3623584Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T22:25:31.3627353Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T22:25:31.3654349Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T22:25:31.3658438Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T22:25:31.3661014Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T22:25:31.3665207Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T22:25:31.3667762Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T22:25:31.3673322Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1) 2024-08-20T22:25:31.3687867Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T22:25:31.4270353Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T22:25:31.4336175Z Building wheels for collected packages: torchtext 2024-08-20T22:26:10.2782588Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:26:10.2815370Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045467 sha256=bbc242b4e74a4f8bb9adffcd69c033e9fd14ba62f4e596835d5a42540b9e2b03 2024-08-20T22:26:10.2817020Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-08-20T22:26:10.2844049Z Successfully built torchtext 2024-08-20T22:26:10.7651441Z Installing collected packages: torchtext 2024-08-20T22:26:10.8803533Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T22:26:11.0442672Z + install_torchvision 2024-08-20T22:26:11.0443084Z + local orig_preload 2024-08-20T22:26:11.0443413Z + local commit 2024-08-20T22:26:11.0445397Z ++ get_pinned_commit vision 2024-08-20T22:26:11.0445894Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:26:11.0464084Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:11.0464547Z + orig_preload= 2024-08-20T22:26:11.0465070Z + '[' -n '' ']' 2024-08-20T22:26:11.0465917Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:11.0467333Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:11.3846198Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:11.3850972Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-v48e74nr 2024-08-20T22:26:11.3881504Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-v48e74nr 2024-08-20T22:26:12.9026796Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:26:12.9053947Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:14.2364798Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:14.5520982Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:17.0865499Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:26:17.0900977Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:26:17.0904633Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:26:17.0909278Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:26:17.0979824Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:26:17.0984634Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:26:17.0986766Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:26:17.0990111Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:26:17.0993452Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:26:17.0999462Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:26:17.1014373Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:26:17.1504476Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:26:17.1566720Z Building wheels for collected packages: torchvision 2024-08-20T22:27:34.8294158Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:27:34.8326937Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116088 sha256=c90c5ffc9bc781b5b53379cd204db948f8808f1fa786530621c89a19714af918 2024-08-20T22:27:34.8328573Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:27:34.8363907Z Successfully built torchvision 2024-08-20T22:27:35.3215587Z Installing collected packages: torchvision 2024-08-20T22:27:35.7374667Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:27:35.8810875Z + '[' -n '' ']' 2024-08-20T22:27:35.8811638Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:27:35.8812194Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T22:27:35.8813394Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T22:27:36.2183392Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T22:27:36.2187801Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-82lxc7pr 2024-08-20T22:27:36.2218227Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-82lxc7pr 2024-08-20T22:27:36.7907396Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:27:39.2394849Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:27:39.2431610Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T22:28:59.0482912Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / done 2024-08-20T22:28:59.0501398Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928017 sha256=ab24ee90a5dbc6eabb39c32d8f13dabdbbb7813b29f100f2957559b87af1812f 2024-08-20T22:28:59.0503557Z Stored in directory: /tmp/pip-ephem-wheel-cache-jng0p36n/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2024-08-20T22:28:59.0534095Z Successfully built torchao 2024-08-20T22:28:59.6128465Z Installing collected packages: torchao 2024-08-20T22:28:59.8538921Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T22:28:59.9659546Z + id=0 2024-08-20T22:28:59.9661389Z + pip_install opencv-python==4.8.0.74 2024-08-20T22:28:59.9662555Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T22:29:00.4311733Z Collecting opencv-python==4.8.0.74 2024-08-20T22:29:00.4477292Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T22:29:00.4556250Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2024-08-20T22:29:00.4617985Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T22:29:01.5741809Z Installing collected packages: opencv-python 2024-08-20T22:29:02.4390573Z Successfully installed opencv-python-4.8.0.74 2024-08-20T22:29:02.5384448Z + [[ aot_inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T22:29:02.5385266Z + [[ aot_inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T22:29:02.5385966Z + [[ aot_inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-08-20T22:29:02.5386489Z + checkout_install_torchbench 2024-08-20T22:29:02.5386889Z + local commit 2024-08-20T22:29:02.5389245Z ++ get_pinned_commit torchbench 2024-08-20T22:29:02.5389707Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T22:29:02.5407693Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:02.5408303Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T22:29:02.5425248Z Cloning into 'torchbench'... 2024-08-20T22:29:02.7046860Z remote: Enumerating objects: 31006, done. 2024-08-20T22:29:02.7047987Z remote: Counting objects: 0% (1/1746) 2024-08-20T22:29:02.7049018Z remote: Counting objects: 1% (18/1746) 2024-08-20T22:29:02.7050773Z remote: Counting objects: 2% (35/1746) 2024-08-20T22:29:02.7051943Z remote: Counting objects: 3% (53/1746) 2024-08-20T22:29:02.7052655Z remote: Counting objects: 4% (70/1746) 2024-08-20T22:29:02.7053194Z remote: Counting objects: 5% (88/1746) 2024-08-20T22:29:02.7053761Z remote: Counting objects: 6% (105/1746) 2024-08-20T22:29:02.7056233Z remote: Counting objects: 7% (123/1746) 2024-08-20T22:29:02.7056952Z remote: Counting objects: 8% (140/1746) 2024-08-20T22:29:02.7057714Z remote: Counting objects: 9% (158/1746) 2024-08-20T22:29:02.7058327Z remote: Counting objects: 10% (175/1746) 2024-08-20T22:29:02.7058916Z remote: Counting objects: 11% (193/1746) 2024-08-20T22:29:02.7059493Z remote: Counting objects: 12% (210/1746) 2024-08-20T22:29:02.7060206Z remote: Counting objects: 13% (227/1746) 2024-08-20T22:29:02.7060921Z remote: Counting objects: 14% (245/1746) 2024-08-20T22:29:02.7061529Z remote: Counting objects: 15% (262/1746) 2024-08-20T22:29:02.7062101Z remote: Counting objects: 16% (280/1746) 2024-08-20T22:29:02.7062811Z remote: Counting objects: 17% (297/1746) 2024-08-20T22:29:02.7063501Z remote: Counting objects: 18% (315/1746) 2024-08-20T22:29:02.7066196Z remote: Counting objects: 19% (332/1746) 2024-08-20T22:29:02.7066820Z remote: Counting objects: 20% (350/1746) 2024-08-20T22:29:02.7067433Z remote: Counting objects: 21% (367/1746) 2024-08-20T22:29:02.7068030Z remote: Counting objects: 22% (385/1746) 2024-08-20T22:29:02.7068641Z remote: Counting objects: 23% (402/1746) 2024-08-20T22:29:02.7069165Z remote: Counting objects: 24% (420/1746) 2024-08-20T22:29:02.7069685Z remote: Counting objects: 25% (437/1746) 2024-08-20T22:29:02.7070199Z remote: Counting objects: 26% (454/1746) 2024-08-20T22:29:02.7070716Z remote: Counting objects: 27% (472/1746) 2024-08-20T22:29:02.7071229Z remote: Counting objects: 28% (489/1746) 2024-08-20T22:29:02.7071765Z remote: Counting objects: 29% (507/1746) 2024-08-20T22:29:02.7072274Z remote: Counting objects: 30% (524/1746) 2024-08-20T22:29:02.7072800Z remote: Counting objects: 31% (542/1746) 2024-08-20T22:29:02.7073719Z remote: Counting objects: 32% (559/1746) 2024-08-20T22:29:02.7074656Z remote: Counting objects: 33% (577/1746) 2024-08-20T22:29:02.7075362Z remote: Counting objects: 34% (594/1746) 2024-08-20T22:29:02.7075893Z remote: Counting objects: 35% (612/1746) 2024-08-20T22:29:02.7076409Z remote: Counting objects: 36% (629/1746) 2024-08-20T22:29:02.7076917Z remote: Counting objects: 37% (647/1746) 2024-08-20T22:29:02.7077419Z remote: Counting objects: 38% (664/1746) 2024-08-20T22:29:02.7077925Z remote: Counting objects: 39% (681/1746) 2024-08-20T22:29:02.7078431Z remote: Counting objects: 40% (699/1746) 2024-08-20T22:29:02.7078930Z remote: Counting objects: 41% (716/1746) 2024-08-20T22:29:02.7079443Z remote: Counting objects: 42% (734/1746) 2024-08-20T22:29:02.7079950Z remote: Counting objects: 43% (751/1746) 2024-08-20T22:29:02.7080462Z remote: Counting objects: 44% (769/1746) 2024-08-20T22:29:02.7080962Z remote: Counting objects: 45% (786/1746) 2024-08-20T22:29:02.7081484Z remote: Counting objects: 46% (804/1746) 2024-08-20T22:29:02.7081999Z remote: Counting objects: 47% (821/1746) 2024-08-20T22:29:02.7082524Z remote: Counting objects: 48% (839/1746) 2024-08-20T22:29:02.7083029Z remote: Counting objects: 49% (856/1746) 2024-08-20T22:29:02.7083540Z remote: Counting objects: 50% (873/1746) 2024-08-20T22:29:02.7084045Z remote: Counting objects: 51% (891/1746) 2024-08-20T22:29:02.7084552Z remote: Counting objects: 52% (908/1746) 2024-08-20T22:29:02.7085054Z remote: Counting objects: 53% (926/1746) 2024-08-20T22:29:02.7085561Z remote: Counting objects: 54% (943/1746) 2024-08-20T22:29:02.7086066Z remote: Counting objects: 55% (961/1746) 2024-08-20T22:29:02.7086574Z remote: Counting objects: 56% (978/1746) 2024-08-20T22:29:02.7087076Z remote: Counting objects: 57% (996/1746) 2024-08-20T22:29:02.7087592Z remote: Counting objects: 58% (1013/1746) 2024-08-20T22:29:02.7088123Z remote: Counting objects: 59% (1031/1746) 2024-08-20T22:29:02.7088638Z remote: Counting objects: 60% (1048/1746) 2024-08-20T22:29:02.7089166Z remote: Counting objects: 61% (1066/1746) 2024-08-20T22:29:02.7089683Z remote: Counting objects: 62% (1083/1746) 2024-08-20T22:29:02.7090203Z remote: Counting objects: 63% (1100/1746) 2024-08-20T22:29:02.7090716Z remote: Counting objects: 64% (1118/1746) 2024-08-20T22:29:02.7091241Z remote: Counting objects: 65% (1135/1746) 2024-08-20T22:29:02.7091756Z remote: Counting objects: 66% (1153/1746) 2024-08-20T22:29:02.7092399Z remote: Counting objects: 67% (1170/1746) 2024-08-20T22:29:02.7092918Z remote: Counting objects: 68% (1188/1746) 2024-08-20T22:29:02.7093438Z remote: Counting objects: 69% (1205/1746) 2024-08-20T22:29:02.7093959Z remote: Counting objects: 70% (1223/1746) 2024-08-20T22:29:02.7094477Z remote: Counting objects: 71% (1240/1746) 2024-08-20T22:29:02.7095002Z remote: Counting objects: 72% (1258/1746) 2024-08-20T22:29:02.7095528Z remote: Counting objects: 73% (1275/1746) 2024-08-20T22:29:02.7096075Z remote: Counting objects: 74% (1293/1746) 2024-08-20T22:29:02.7096600Z remote: Counting objects: 75% (1310/1746) 2024-08-20T22:29:02.7097108Z remote: Counting objects: 76% (1327/1746) 2024-08-20T22:29:02.7097969Z remote: Counting objects: 77% (1345/1746) 2024-08-20T22:29:02.7098491Z remote: Counting objects: 78% (1362/1746) 2024-08-20T22:29:02.7099013Z remote: Counting objects: 79% (1380/1746) 2024-08-20T22:29:02.7099527Z remote: Counting objects: 80% (1397/1746) 2024-08-20T22:29:02.7100043Z remote: Counting objects: 81% (1415/1746) 2024-08-20T22:29:02.7100559Z remote: Counting objects: 82% (1432/1746) 2024-08-20T22:29:02.7101077Z remote: Counting objects: 83% (1450/1746) 2024-08-20T22:29:02.7101589Z remote: Counting objects: 84% (1467/1746) 2024-08-20T22:29:02.7102104Z remote: Counting objects: 85% (1485/1746) 2024-08-20T22:29:02.7102620Z remote: Counting objects: 86% (1502/1746) 2024-08-20T22:29:02.7103299Z remote: Counting objects: 87% (1520/1746) 2024-08-20T22:29:02.7103923Z remote: Counting objects: 88% (1537/1746) 2024-08-20T22:29:02.7104447Z remote: Counting objects: 89% (1554/1746) 2024-08-20T22:29:02.7104965Z remote: Counting objects: 90% (1572/1746) 2024-08-20T22:29:02.7105470Z remote: Counting objects: 91% (1589/1746) 2024-08-20T22:29:02.7105990Z remote: Counting objects: 92% (1607/1746) 2024-08-20T22:29:02.7106502Z remote: Counting objects: 93% (1624/1746) 2024-08-20T22:29:02.7107016Z remote: Counting objects: 94% (1642/1746) 2024-08-20T22:29:02.7107527Z remote: Counting objects: 95% (1659/1746) 2024-08-20T22:29:02.7108040Z remote: Counting objects: 96% (1677/1746) 2024-08-20T22:29:02.7108552Z remote: Counting objects: 97% (1694/1746) 2024-08-20T22:29:02.7109069Z remote: Counting objects: 98% (1712/1746) 2024-08-20T22:29:02.7109579Z remote: Counting objects: 99% (1729/1746) 2024-08-20T22:29:02.7110101Z remote: Counting objects: 100% (1746/1746) 2024-08-20T22:29:02.7110653Z remote: Counting objects: 100% (1746/1746), done. 2024-08-20T22:29:02.7111222Z remote: Compressing objects: 0% (1/904) 2024-08-20T22:29:02.7153117Z remote: Compressing objects: 1% (10/904) 2024-08-20T22:29:02.7268273Z remote: Compressing objects: 2% (19/904) 2024-08-20T22:29:02.7351868Z remote: Compressing objects: 3% (28/904) 2024-08-20T22:29:02.7401043Z remote: Compressing objects: 4% (37/904) 2024-08-20T22:29:02.7448981Z remote: Compressing objects: 5% (46/904) 2024-08-20T22:29:02.7484462Z remote: Compressing objects: 6% (55/904) 2024-08-20T22:29:02.7512305Z remote: Compressing objects: 7% (64/904) 2024-08-20T22:29:02.7545469Z remote: Compressing objects: 8% (73/904) 2024-08-20T22:29:02.7583798Z remote: Compressing objects: 9% (82/904) 2024-08-20T22:29:02.7608287Z remote: Compressing objects: 10% (91/904) 2024-08-20T22:29:02.7624731Z remote: Compressing objects: 11% (100/904) 2024-08-20T22:29:02.7635735Z remote: Compressing objects: 12% (109/904) 2024-08-20T22:29:02.7650603Z remote: Compressing objects: 13% (118/904) 2024-08-20T22:29:02.7667104Z remote: Compressing objects: 14% (127/904) 2024-08-20T22:29:02.7676219Z remote: Compressing objects: 15% (136/904) 2024-08-20T22:29:02.7684335Z remote: Compressing objects: 16% (145/904) 2024-08-20T22:29:02.7690562Z remote: Compressing objects: 17% (154/904) 2024-08-20T22:29:02.7693724Z remote: Compressing objects: 18% (163/904) 2024-08-20T22:29:02.7696660Z remote: Compressing objects: 19% (172/904) 2024-08-20T22:29:02.7699209Z remote: Compressing objects: 20% (181/904) 2024-08-20T22:29:02.7700524Z remote: Compressing objects: 21% (190/904) 2024-08-20T22:29:02.7701070Z remote: Compressing objects: 22% (199/904) 2024-08-20T22:29:02.7703720Z remote: Compressing objects: 23% (208/904) 2024-08-20T22:29:02.7704486Z remote: Compressing objects: 24% (217/904) 2024-08-20T22:29:02.7706436Z remote: Compressing objects: 25% (226/904) 2024-08-20T22:29:02.7707705Z remote: Compressing objects: 26% (236/904) 2024-08-20T22:29:02.7715857Z remote: Compressing objects: 27% (245/904) 2024-08-20T22:29:02.7718960Z remote: Compressing objects: 28% (254/904) 2024-08-20T22:29:02.7722336Z remote: Compressing objects: 29% (263/904) 2024-08-20T22:29:02.7724829Z remote: Compressing objects: 30% (272/904) 2024-08-20T22:29:02.7727495Z remote: Compressing objects: 31% (281/904) 2024-08-20T22:29:02.7730258Z remote: Compressing objects: 32% (290/904) 2024-08-20T22:29:02.7733205Z remote: Compressing objects: 33% (299/904) 2024-08-20T22:29:02.7736689Z remote: Compressing objects: 34% (308/904) 2024-08-20T22:29:02.7738999Z remote: Compressing objects: 35% (317/904) 2024-08-20T22:29:02.7741481Z remote: Compressing objects: 36% (326/904) 2024-08-20T22:29:02.7744257Z remote: Compressing objects: 37% (335/904) 2024-08-20T22:29:02.7746215Z remote: Compressing objects: 38% (344/904) 2024-08-20T22:29:02.7748183Z remote: Compressing objects: 39% (353/904) 2024-08-20T22:29:02.7749487Z remote: Compressing objects: 40% (362/904) 2024-08-20T22:29:02.7751733Z remote: Compressing objects: 41% (371/904) 2024-08-20T22:29:02.7752370Z remote: Compressing objects: 42% (380/904) 2024-08-20T22:29:02.7753428Z remote: Compressing objects: 43% (389/904) 2024-08-20T22:29:02.7754344Z remote: Compressing objects: 44% (398/904) 2024-08-20T22:29:02.7755363Z remote: Compressing objects: 45% (407/904) 2024-08-20T22:29:02.7756099Z remote: Compressing objects: 46% (416/904) 2024-08-20T22:29:02.7756740Z remote: Compressing objects: 47% (425/904) 2024-08-20T22:29:02.7757390Z remote: Compressing objects: 48% (434/904) 2024-08-20T22:29:02.7758141Z remote: Compressing objects: 49% (443/904) 2024-08-20T22:29:02.7758794Z remote: Compressing objects: 50% (452/904) 2024-08-20T22:29:02.7759543Z remote: Compressing objects: 51% (462/904) 2024-08-20T22:29:02.7760250Z remote: Compressing objects: 52% (471/904) 2024-08-20T22:29:02.7760900Z remote: Compressing objects: 53% (480/904) 2024-08-20T22:29:02.7761592Z remote: Compressing objects: 54% (489/904) 2024-08-20T22:29:02.7762275Z remote: Compressing objects: 55% (498/904) 2024-08-20T22:29:02.7762913Z remote: Compressing objects: 56% (507/904) 2024-08-20T22:29:02.7763548Z remote: Compressing objects: 57% (516/904) 2024-08-20T22:29:02.7764306Z remote: Compressing objects: 58% (525/904) 2024-08-20T22:29:02.7764851Z remote: Compressing objects: 59% (534/904) 2024-08-20T22:29:02.7765455Z remote: Compressing objects: 60% (543/904) 2024-08-20T22:29:02.7765991Z remote: Compressing objects: 61% (552/904) 2024-08-20T22:29:02.7766524Z remote: Compressing objects: 62% (561/904) 2024-08-20T22:29:02.7767057Z remote: Compressing objects: 63% (570/904) 2024-08-20T22:29:02.7767590Z remote: Compressing objects: 64% (579/904) 2024-08-20T22:29:02.7768192Z remote: Compressing objects: 65% (588/904) 2024-08-20T22:29:02.7768809Z remote: Compressing objects: 66% (597/904) 2024-08-20T22:29:02.7769424Z remote: Compressing objects: 67% (606/904) 2024-08-20T22:29:02.7769957Z remote: Compressing objects: 68% (615/904) 2024-08-20T22:29:02.7770488Z remote: Compressing objects: 69% (624/904) 2024-08-20T22:29:02.7771024Z remote: Compressing objects: 70% (633/904) 2024-08-20T22:29:02.7771553Z remote: Compressing objects: 71% (642/904) 2024-08-20T22:29:02.7772077Z remote: Compressing objects: 72% (651/904) 2024-08-20T22:29:02.7772746Z remote: Compressing objects: 73% (660/904) 2024-08-20T22:29:02.7773295Z remote: Compressing objects: 74% (669/904) 2024-08-20T22:29:02.7773861Z remote: Compressing objects: 75% (678/904) 2024-08-20T22:29:02.7777867Z remote: Compressing objects: 76% (688/904) 2024-08-20T22:29:02.7787155Z remote: Compressing objects: 77% (697/904) 2024-08-20T22:29:02.7789998Z remote: Compressing objects: 78% (706/904) 2024-08-20T22:29:02.7796085Z remote: Compressing objects: 79% (715/904) 2024-08-20T22:29:02.7802959Z remote: Compressing objects: 80% (724/904) 2024-08-20T22:29:02.7811751Z remote: Compressing objects: 81% (733/904) 2024-08-20T22:29:02.7818623Z remote: Compressing objects: 82% (742/904) 2024-08-20T22:29:02.7821918Z remote: Compressing objects: 83% (751/904) 2024-08-20T22:29:02.7825608Z remote: Compressing objects: 84% (760/904) 2024-08-20T22:29:02.7829687Z remote: Compressing objects: 85% (769/904) 2024-08-20T22:29:02.7834773Z remote: Compressing objects: 86% (778/904) 2024-08-20T22:29:02.7838824Z remote: Compressing objects: 87% (787/904) 2024-08-20T22:29:02.7842244Z remote: Compressing objects: 88% (796/904) 2024-08-20T22:29:02.7846759Z remote: Compressing objects: 89% (805/904) 2024-08-20T22:29:02.7849839Z remote: Compressing objects: 90% (814/904) 2024-08-20T22:29:02.7852625Z remote: Compressing objects: 91% (823/904) 2024-08-20T22:29:02.7856114Z remote: Compressing objects: 92% (832/904) 2024-08-20T22:29:02.7858897Z remote: Compressing objects: 93% (841/904) 2024-08-20T22:29:02.7861661Z remote: Compressing objects: 94% (850/904) 2024-08-20T22:29:02.7863820Z remote: Compressing objects: 95% (859/904) 2024-08-20T22:29:02.7866761Z remote: Compressing objects: 96% (868/904) 2024-08-20T22:29:02.7869196Z remote: Compressing objects: 97% (877/904) 2024-08-20T22:29:02.7871009Z remote: Compressing objects: 98% (886/904) 2024-08-20T22:29:02.7871619Z remote: Compressing objects: 99% (895/904) 2024-08-20T22:29:02.7872570Z remote: Compressing objects: 100% (904/904) 2024-08-20T22:29:02.7873257Z remote: Compressing objects: 100% (904/904), done. 2024-08-20T22:29:02.7952506Z Receiving objects: 0% (1/31006) 2024-08-20T22:29:02.7994374Z Receiving objects: 1% (311/31006) 2024-08-20T22:29:02.8034698Z Receiving objects: 2% (621/31006) 2024-08-20T22:29:02.8070687Z Receiving objects: 3% (931/31006) 2024-08-20T22:29:02.8104533Z Receiving objects: 4% (1241/31006) 2024-08-20T22:29:02.8137456Z Receiving objects: 5% (1551/31006) 2024-08-20T22:29:02.8174482Z Receiving objects: 6% (1861/31006) 2024-08-20T22:29:02.8214179Z Receiving objects: 7% (2171/31006) 2024-08-20T22:29:02.8244155Z Receiving objects: 8% (2481/31006) 2024-08-20T22:29:02.8277142Z Receiving objects: 9% (2791/31006) 2024-08-20T22:29:02.8347601Z Receiving objects: 10% (3101/31006) 2024-08-20T22:29:02.8395968Z Receiving objects: 11% (3411/31006) 2024-08-20T22:29:02.8462251Z Receiving objects: 12% (3721/31006) 2024-08-20T22:29:02.8593345Z Receiving objects: 13% (4031/31006) 2024-08-20T22:29:02.8739852Z Receiving objects: 14% (4341/31006) 2024-08-20T22:29:02.8811324Z Receiving objects: 15% (4651/31006) 2024-08-20T22:29:02.8884259Z Receiving objects: 16% (4961/31006) 2024-08-20T22:29:02.8943762Z Receiving objects: 17% (5272/31006) 2024-08-20T22:29:02.8971122Z Receiving objects: 18% (5582/31006) 2024-08-20T22:29:02.9030429Z Receiving objects: 19% (5892/31006) 2024-08-20T22:29:02.9063600Z Receiving objects: 20% (6202/31006) 2024-08-20T22:29:02.9078082Z Receiving objects: 21% (6512/31006) 2024-08-20T22:29:02.9278358Z Receiving objects: 22% (6822/31006) 2024-08-20T22:29:03.7900369Z Receiving objects: 23% (7132/31006) 2024-08-20T22:29:03.9079886Z Receiving objects: 23% (7376/31006), 75.63 MiB | 75.70 MiB/s 2024-08-20T22:29:04.4590683Z Receiving objects: 24% (7442/31006), 75.63 MiB | 75.70 MiB/s 2024-08-20T22:29:04.5311063Z Receiving objects: 25% (7752/31006), 113.26 MiB | 75.55 MiB/s 2024-08-20T22:29:04.6037385Z Receiving objects: 26% (8062/31006), 113.26 MiB | 75.55 MiB/s 2024-08-20T22:29:04.6750372Z Receiving objects: 27% (8372/31006), 113.26 MiB | 75.55 MiB/s 2024-08-20T22:29:04.7464394Z Receiving objects: 28% (8682/31006), 113.26 MiB | 75.55 MiB/s 2024-08-20T22:29:04.7898731Z Receiving objects: 29% (8992/31006), 113.26 MiB | 75.55 MiB/s 2024-08-20T22:29:04.8184801Z Receiving objects: 29% (9181/31006), 113.26 MiB | 75.55 MiB/s 2024-08-20T22:29:04.8903979Z Receiving objects: 30% (9302/31006), 151.55 MiB | 75.81 MiB/s 2024-08-20T22:29:04.9619967Z Receiving objects: 31% (9612/31006), 151.55 MiB | 75.81 MiB/s 2024-08-20T22:29:05.1954586Z Receiving objects: 32% (9922/31006), 151.55 MiB | 75.81 MiB/s 2024-08-20T22:29:05.2694910Z Receiving objects: 33% (10232/31006), 151.55 MiB | 75.81 MiB/s 2024-08-20T22:29:05.2819139Z Receiving objects: 34% (10543/31006), 151.55 MiB | 75.81 MiB/s 2024-08-20T22:29:05.5414328Z Receiving objects: 35% (10853/31006), 151.55 MiB | 75.81 MiB/s 2024-08-20T22:29:05.7902503Z Receiving objects: 36% (11163/31006), 184.87 MiB | 73.98 MiB/s 2024-08-20T22:29:05.7973660Z Receiving objects: 36% (11449/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.7992365Z Receiving objects: 37% (11473/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8015913Z Receiving objects: 38% (11783/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8034547Z Receiving objects: 39% (12093/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8050226Z Receiving objects: 40% (12403/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8061560Z Receiving objects: 41% (12713/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8074257Z Receiving objects: 42% (13023/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8090142Z Receiving objects: 43% (13333/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8107356Z Receiving objects: 44% (13643/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8115754Z Receiving objects: 45% (13953/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8121004Z Receiving objects: 46% (14263/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8135708Z Receiving objects: 47% (14573/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8157033Z Receiving objects: 48% (14883/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8170323Z Receiving objects: 49% (15193/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8371508Z Receiving objects: 50% (15503/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8378164Z Receiving objects: 51% (15814/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8385180Z Receiving objects: 52% (16124/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8392172Z Receiving objects: 53% (16434/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8398051Z Receiving objects: 54% (16744/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8416349Z Receiving objects: 55% (17054/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8427293Z Receiving objects: 56% (17364/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8434031Z Receiving objects: 57% (17674/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8448047Z Receiving objects: 58% (17984/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8460619Z Receiving objects: 59% (18294/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8471158Z Receiving objects: 60% (18604/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8608912Z Receiving objects: 61% (18914/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8615479Z Receiving objects: 62% (19224/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8621805Z Receiving objects: 63% (19534/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8630430Z Receiving objects: 64% (19844/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8638526Z Receiving objects: 65% (20154/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8648035Z Receiving objects: 66% (20464/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.8903503Z Receiving objects: 67% (20775/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9300889Z Receiving objects: 68% (21085/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9309620Z Receiving objects: 69% (21395/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9323141Z Receiving objects: 70% (21705/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9334883Z Receiving objects: 71% (22015/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9349795Z Receiving objects: 72% (22325/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9359553Z Receiving objects: 73% (22635/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9376465Z Receiving objects: 74% (22945/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9386750Z Receiving objects: 75% (23255/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9394563Z Receiving objects: 76% (23565/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9403711Z Receiving objects: 77% (23875/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9617420Z Receiving objects: 78% (24185/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9651135Z Receiving objects: 79% (24495/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9660795Z Receiving objects: 80% (24805/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9673378Z Receiving objects: 81% (25115/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9686153Z Receiving objects: 82% (25425/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9735124Z Receiving objects: 83% (25735/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9873198Z Receiving objects: 84% (26046/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9879678Z Receiving objects: 85% (26356/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9888128Z Receiving objects: 86% (26666/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9893749Z Receiving objects: 87% (26976/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9902712Z Receiving objects: 88% (27286/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:05.9908889Z Receiving objects: 89% (27596/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0021378Z Receiving objects: 90% (27906/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0035815Z Receiving objects: 91% (28216/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0048254Z Receiving objects: 92% (28526/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0066179Z Receiving objects: 93% (28836/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0101023Z Receiving objects: 94% (29146/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0112712Z Receiving objects: 95% (29456/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0120922Z Receiving objects: 96% (29766/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0126691Z Receiving objects: 97% (30076/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0198186Z Receiving objects: 98% (30386/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0222577Z Receiving objects: 99% (30696/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0224045Z remote: Total 31006 (delta 1013), reused 1476 (delta 823), pack-reused 29260 (from 1) 2024-08-20T22:29:06.0240741Z Receiving objects: 100% (31006/31006), 221.42 MiB | 73.83 MiB/s 2024-08-20T22:29:06.0241704Z Receiving objects: 100% (31006/31006), 232.50 MiB | 71.91 MiB/s, done. 2024-08-20T22:29:06.0273221Z Resolving deltas: 0% (0/16726) 2024-08-20T22:29:06.0286728Z Resolving deltas: 1% (168/16726) 2024-08-20T22:29:06.0297752Z Resolving deltas: 2% (397/16726) 2024-08-20T22:29:06.0320592Z Resolving deltas: 3% (505/16726) 2024-08-20T22:29:06.0336491Z Resolving deltas: 4% (671/16726) 2024-08-20T22:29:06.0346010Z Resolving deltas: 5% (900/16726) 2024-08-20T22:29:06.0355296Z Resolving deltas: 6% (1021/16726) 2024-08-20T22:29:06.0368641Z Resolving deltas: 7% (1171/16726) 2024-08-20T22:29:06.0377474Z Resolving deltas: 8% (1351/16726) 2024-08-20T22:29:06.0388911Z Resolving deltas: 9% (1517/16726) 2024-08-20T22:29:06.0400744Z Resolving deltas: 10% (1677/16726) 2024-08-20T22:29:06.0413738Z Resolving deltas: 11% (1845/16726) 2024-08-20T22:29:06.0426566Z Resolving deltas: 12% (2008/16726) 2024-08-20T22:29:06.0439467Z Resolving deltas: 13% (2189/16726) 2024-08-20T22:29:06.0452520Z Resolving deltas: 14% (2393/16726) 2024-08-20T22:29:06.0468073Z Resolving deltas: 15% (2511/16726) 2024-08-20T22:29:06.0484476Z Resolving deltas: 16% (2680/16726) 2024-08-20T22:29:06.0497627Z Resolving deltas: 17% (2865/16726) 2024-08-20T22:29:06.0547242Z Resolving deltas: 18% (3021/16726) 2024-08-20T22:29:06.0568900Z Resolving deltas: 19% (3197/16726) 2024-08-20T22:29:06.0571370Z Resolving deltas: 20% (3484/16726) 2024-08-20T22:29:06.0585269Z Resolving deltas: 21% (3514/16726) 2024-08-20T22:29:06.0602499Z Resolving deltas: 22% (3694/16726) 2024-08-20T22:29:06.0613485Z Resolving deltas: 23% (3898/16726) 2024-08-20T22:29:06.0627929Z Resolving deltas: 24% (4043/16726) 2024-08-20T22:29:06.0665832Z Resolving deltas: 25% (4192/16726) 2024-08-20T22:29:06.0686320Z Resolving deltas: 26% (4360/16726) 2024-08-20T22:29:06.0701790Z Resolving deltas: 27% (4533/16726) 2024-08-20T22:29:06.0719828Z Resolving deltas: 28% (4685/16726) 2024-08-20T22:29:06.0737514Z Resolving deltas: 29% (4851/16726) 2024-08-20T22:29:06.0750620Z Resolving deltas: 30% (5041/16726) 2024-08-20T22:29:06.0761314Z Resolving deltas: 31% (5189/16726) 2024-08-20T22:29:06.0772305Z Resolving deltas: 32% (5355/16726) 2024-08-20T22:29:06.0785915Z Resolving deltas: 33% (5521/16726) 2024-08-20T22:29:06.0807383Z Resolving deltas: 34% (5692/16726) 2024-08-20T22:29:06.0828543Z Resolving deltas: 35% (5858/16726) 2024-08-20T22:29:06.0842607Z Resolving deltas: 36% (6044/16726) 2024-08-20T22:29:06.0852807Z Resolving deltas: 37% (6235/16726) 2024-08-20T22:29:06.0871803Z Resolving deltas: 38% (6356/16726) 2024-08-20T22:29:06.0885595Z Resolving deltas: 39% (6524/16726) 2024-08-20T22:29:06.0904042Z Resolving deltas: 40% (6709/16726) 2024-08-20T22:29:06.0917585Z Resolving deltas: 41% (6891/16726) 2024-08-20T22:29:06.0927666Z Resolving deltas: 42% (7036/16726) 2024-08-20T22:29:06.0937698Z Resolving deltas: 43% (7227/16726) 2024-08-20T22:29:06.0952668Z Resolving deltas: 44% (7367/16726) 2024-08-20T22:29:06.0978351Z Resolving deltas: 45% (7527/16726) 2024-08-20T22:29:06.0986880Z Resolving deltas: 46% (7702/16726) 2024-08-20T22:29:06.1007001Z Resolving deltas: 47% (7872/16726) 2024-08-20T22:29:06.1015652Z Resolving deltas: 48% (8030/16726) 2024-08-20T22:29:06.1025016Z Resolving deltas: 49% (8196/16726) 2024-08-20T22:29:06.1034889Z Resolving deltas: 50% (8372/16726) 2024-08-20T22:29:06.1064469Z Resolving deltas: 51% (8532/16726) 2024-08-20T22:29:06.1070772Z Resolving deltas: 53% (8901/16726) 2024-08-20T22:29:06.1080000Z Resolving deltas: 54% (9033/16726) 2024-08-20T22:29:06.1095824Z Resolving deltas: 55% (9200/16726) 2024-08-20T22:29:06.1104798Z Resolving deltas: 56% (9370/16726) 2024-08-20T22:29:06.1114549Z Resolving deltas: 57% (9569/16726) 2024-08-20T22:29:06.1118108Z Resolving deltas: 58% (9806/16726) 2024-08-20T22:29:06.1133893Z Resolving deltas: 59% (9871/16726) 2024-08-20T22:29:06.1143193Z Resolving deltas: 60% (10098/16726) 2024-08-20T22:29:06.1159523Z Resolving deltas: 61% (10268/16726) 2024-08-20T22:29:06.1163833Z Resolving deltas: 62% (10468/16726) 2024-08-20T22:29:06.1178443Z Resolving deltas: 63% (10543/16726) 2024-08-20T22:29:06.1206551Z Resolving deltas: 64% (10723/16726) 2024-08-20T22:29:06.1220247Z Resolving deltas: 65% (11016/16726) 2024-08-20T22:29:06.1244198Z Resolving deltas: 67% (11212/16726) 2024-08-20T22:29:06.1253629Z Resolving deltas: 68% (11400/16726) 2024-08-20T22:29:06.1265556Z Resolving deltas: 69% (11591/16726) 2024-08-20T22:29:06.1280376Z Resolving deltas: 70% (11751/16726) 2024-08-20T22:29:06.1293199Z Resolving deltas: 71% (11877/16726) 2024-08-20T22:29:06.1304137Z Resolving deltas: 72% (12046/16726) 2024-08-20T22:29:06.1312182Z Resolving deltas: 73% (12214/16726) 2024-08-20T22:29:06.1328306Z Resolving deltas: 74% (12379/16726) 2024-08-20T22:29:06.1333242Z Resolving deltas: 75% (12641/16726) 2024-08-20T22:29:06.1340781Z Resolving deltas: 76% (12730/16726) 2024-08-20T22:29:06.1349555Z Resolving deltas: 77% (12887/16726) 2024-08-20T22:29:06.1361048Z Resolving deltas: 78% (13050/16726) 2024-08-20T22:29:06.1370438Z Resolving deltas: 79% (13215/16726) 2024-08-20T22:29:06.1380052Z Resolving deltas: 80% (13381/16726) 2024-08-20T22:29:06.1389988Z Resolving deltas: 81% (13567/16726) 2024-08-20T22:29:06.1400928Z Resolving deltas: 82% (13739/16726) 2024-08-20T22:29:06.1409778Z Resolving deltas: 83% (13900/16726) 2024-08-20T22:29:06.1416428Z Resolving deltas: 84% (14079/16726) 2024-08-20T22:29:06.1424657Z Resolving deltas: 85% (14219/16726) 2024-08-20T22:29:06.1434381Z Resolving deltas: 86% (14393/16726) 2024-08-20T22:29:06.1442431Z Resolving deltas: 87% (14552/16726) 2024-08-20T22:29:06.1452607Z Resolving deltas: 88% (14720/16726) 2024-08-20T22:29:06.1459387Z Resolving deltas: 89% (14899/16726) 2024-08-20T22:29:06.1467847Z Resolving deltas: 90% (15068/16726) 2024-08-20T22:29:06.1476872Z Resolving deltas: 91% (15222/16726) 2024-08-20T22:29:06.1484252Z Resolving deltas: 92% (15390/16726) 2024-08-20T22:29:06.1494743Z Resolving deltas: 93% (15557/16726) 2024-08-20T22:29:06.1504605Z Resolving deltas: 94% (15728/16726) 2024-08-20T22:29:06.1511361Z Resolving deltas: 95% (15920/16726) 2024-08-20T22:29:06.1519639Z Resolving deltas: 96% (16074/16726) 2024-08-20T22:29:06.1528026Z Resolving deltas: 97% (16247/16726) 2024-08-20T22:29:06.1573397Z Resolving deltas: 98% (16392/16726) 2024-08-20T22:29:06.1647966Z Resolving deltas: 99% (16562/16726) 2024-08-20T22:29:06.1648407Z Resolving deltas: 100% (16726/16726) 2024-08-20T22:29:06.1649062Z Resolving deltas: 100% (16726/16726), done. 2024-08-20T22:29:07.1653941Z + pushd torchbench 2024-08-20T22:29:07.1654859Z ~/workspace/torchbench ~/workspace 2024-08-20T22:29:07.1655433Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:07.2430513Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T22:29:07.2431095Z 2024-08-20T22:29:07.2431636Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T22:29:07.2432560Z changes and commit them, and you can discard any commits you make in this 2024-08-20T22:29:07.2433322Z state without impacting any branches by switching back to a branch. 2024-08-20T22:29:07.2433755Z 2024-08-20T22:29:07.2434069Z If you want to create a new branch to retain commits you create, you may 2024-08-20T22:29:07.2434868Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T22:29:07.2435288Z 2024-08-20T22:29:07.2435490Z git switch -c 2024-08-20T22:29:07.2435780Z 2024-08-20T22:29:07.2435917Z Or undo this operation with: 2024-08-20T22:29:07.2436160Z 2024-08-20T22:29:07.2436289Z git switch - 2024-08-20T22:29:07.2436468Z 2024-08-20T22:29:07.2436806Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T22:29:07.2437285Z 2024-08-20T22:29:07.2437448Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T22:29:07.2441558Z + '[' '' ']' 2024-08-20T22:29:07.2442058Z + python install.py --continue_on_fail 2024-08-20T22:29:09.5228170Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T22:29:27.7766954Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T22:29:30.9369799Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T22:29:32.8386188Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T22:29:34.4729765Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T22:29:34.4975298Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T22:29:42.1605464Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T22:29:44.9014659Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T22:29:47.6271933Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T22:29:50.3532491Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T22:29:50.3538977Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T22:29:51.4046038Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T22:29:54.1721228Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T22:29:54.1958070Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T22:30:51.2693424Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T22:30:59.8590179Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T22:31:07.1699972Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T22:31:13.9904139Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T22:31:22.6801606Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T22:31:29.7310601Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T22:31:36.1114717Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T22:31:43.1510366Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T22:31:50.1684488Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T22:31:57.3061717Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T22:32:04.0784963Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T22:32:11.1257409Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T22:32:12.3311603Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T22:32:49.3140501Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T22:32:53.8445098Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T22:32:59.0859077Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:33:10.2397839Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:33:11.4367143Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:33:12.9050501Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:33:19.5216304Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:33:26.6056523Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:33:32.1040628Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:33:39.8228773Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:33:45.7219536Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:33:50.7569107Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:33:56.9578861Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:34:13.3972928Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:34:19.5802720Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:34:23.5199909Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:34:28.3552228Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:34:35.1579881Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:34:35.1582308Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:34:48.1640370Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:34:52.1000859Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:34:52.1003312Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:34:59.7176678Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:35:00.9447919Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:35:02.1993533Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:35:47.0905702Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:37:16.1630318Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:37:16.1915435Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:37:17.7360655Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:37:17.7604449Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:37:17.7840455Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:37:17.8084853Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:37:17.8319473Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:37:17.8563377Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:37:38.9812022Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:37:38.9814610Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:37:40.2335161Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:37:42.6661097Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:37:42.6903536Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:37:42.7148939Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:37:42.7397818Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:37:42.7647361Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:37:42.7899518Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:37:49.4354867Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:37:50.8848755Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:37:54.3773684Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:37:54.4016725Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:37:54.4256968Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:37:54.4500599Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:37:54.4746695Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:37:54.4984681Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:38:09.7565573Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:38:26.9112532Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:38:26.9356738Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:38:26.9360074Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:38:26.9361804Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:38:29.3708892Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:38:31.1449505Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:38:31.1696860Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:38:54.8127750Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:39:01.4500012Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:39:05.9603307Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:39:12.9963081Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:39:13.0207367Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:39:13.0450362Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:39:13.0699869Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:39:13.0951413Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:39:13.1190941Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:39:13.1432242Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:39:13.1672230Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:39:22.4196802Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:39:34.2320087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:39:34.2571668Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:39:36.3584170Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:39:38.3610891Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:39:38.8662487Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:39:38.8663138Z Print all dependencies after TorchBench is installed 2024-08-20T22:39:38.8663670Z + python -mpip freeze 2024-08-20T22:39:39.3660683Z absl-py==2.1.0 2024-08-20T22:39:39.3661160Z accelerate==0.33.0 2024-08-20T22:39:39.3661527Z aiohappyeyeballs==2.4.0 2024-08-20T22:39:39.3661889Z aiohttp==3.10.5 2024-08-20T22:39:39.3662262Z aiosignal==1.3.1 2024-08-20T22:39:39.3662672Z annotated-types==0.7.0 2024-08-20T22:39:39.3669384Z antlr4-python3-runtime==4.9.3 2024-08-20T22:39:39.3669964Z anyascii==0.3.2 2024-08-20T22:39:39.3670415Z astroid==3.2.4 2024-08-20T22:39:39.3670775Z astunparse==1.6.3 2024-08-20T22:39:39.3671302Z async-timeout==4.0.3 2024-08-20T22:39:39.3671723Z attrs==23.1.0 2024-08-20T22:39:39.3672059Z audioread==3.0.1 2024-08-20T22:39:39.3672371Z babel==2.16.0 2024-08-20T22:39:39.3673443Z beautifulsoup4==4.12.3 2024-08-20T22:39:39.3674891Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:39:39.3676212Z bibtexparser==2.0.0b7 2024-08-20T22:39:39.3676548Z black==24.8.0 2024-08-20T22:39:39.3676918Z blinker==1.8.2 2024-08-20T22:39:39.3677306Z blis==0.7.11 2024-08-20T22:39:39.3677670Z blobfile==2.1.1 2024-08-20T22:39:39.3678062Z bokeh==3.5.1 2024-08-20T22:39:39.3678431Z boto3==1.19.12 2024-08-20T22:39:39.3678807Z botocore==1.22.12 2024-08-20T22:39:39.3679206Z bs4==0.0.2 2024-08-20T22:39:39.3679567Z cachetools==5.5.0 2024-08-20T22:39:39.3679965Z cardboardlint==1.3.1 2024-08-20T22:39:39.3680877Z catalogue==2.0.10 2024-08-20T22:39:39.3681241Z certifi==2024.7.4 2024-08-20T22:39:39.3681561Z cffi==1.17.0 2024-08-20T22:39:39.3682289Z charset-normalizer==3.3.2 2024-08-20T22:39:39.3682762Z click==8.1.7 2024-08-20T22:39:39.3683149Z clldutils==3.22.2 2024-08-20T22:39:39.3683568Z cloudpathlib==0.18.1 2024-08-20T22:39:39.3683969Z cloudpickle==3.0.0 2024-08-20T22:39:39.3684395Z colorama==0.4.6 2024-08-20T22:39:39.3684783Z colorlog==6.8.2 2024-08-20T22:39:39.3685187Z confection==0.1.5 2024-08-20T22:39:39.3685569Z contourpy==1.2.1 2024-08-20T22:39:39.3685966Z coremltools==5.0b5 2024-08-20T22:39:39.3686379Z cryptography==43.0.0 2024-08-20T22:39:39.3686784Z csvw==3.3.0 2024-08-20T22:39:39.3687129Z cycler==0.12.1 2024-08-20T22:39:39.3687420Z cymem==2.0.8 2024-08-20T22:39:39.3687765Z DALL-E==0.1 2024-08-20T22:39:39.3688057Z datasets==2.21.0 2024-08-20T22:39:39.3688366Z decorator==5.1.1 2024-08-20T22:39:39.3688665Z defusedxml==0.7.1 2024-08-20T22:39:39.3688968Z Deprecated==1.2.14 2024-08-20T22:39:39.3689670Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:39:39.3690458Z diffusers==0.20.2 2024-08-20T22:39:39.3690758Z dill==0.3.8 2024-08-20T22:39:39.3691037Z diskcache==5.6.3 2024-08-20T22:39:39.3691324Z distro==1.9.0 2024-08-20T22:39:39.3691616Z dlinfo==1.2.1 2024-08-20T22:39:39.3691954Z docker-pycreds==0.4.0 2024-08-20T22:39:39.3692281Z dominate==2.9.1 2024-08-20T22:39:39.3693209Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:39:39.3693985Z einops==0.8.0 2024-08-20T22:39:39.3694294Z exceptiongroup==1.2.2 2024-08-20T22:39:39.3694618Z execnet==2.1.1 2024-08-20T22:39:39.3694911Z expecttest==0.1.6 2024-08-20T22:39:39.3695212Z FastNLP==0.6.0 2024-08-20T22:39:39.3695545Z ffmpeg-python==0.2.0 2024-08-20T22:39:39.3695866Z filelock==3.13.1 2024-08-20T22:39:39.3696165Z Flask==3.0.3 2024-08-20T22:39:39.3696457Z flatbuffers==2.0 2024-08-20T22:39:39.3696752Z fonttools==4.53.1 2024-08-20T22:39:39.3697061Z frozenlist==1.4.1 2024-08-20T22:39:39.3697359Z fsspec==2024.6.1 2024-08-20T22:39:39.3698030Z ftfy==6.2.3 2024-08-20T22:39:39.3698637Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:39:39.3699339Z future==1.0.0 2024-08-20T22:39:39.3699636Z fvcore==0.1.5.post20221221 2024-08-20T22:39:39.3699967Z gdown==5.2.0 2024-08-20T22:39:39.3700247Z geojson==2.5.0 2024-08-20T22:39:39.3700537Z ghstack==0.8.0 2024-08-20T22:39:39.3700827Z gitdb==4.0.11 2024-08-20T22:39:39.3701110Z GitPython==3.1.43 2024-08-20T22:39:39.3701516Z google-auth==2.34.0 2024-08-20T22:39:39.3701893Z google-auth-oauthlib==1.0.0 2024-08-20T22:39:39.3702245Z grpcio==1.65.5 2024-08-20T22:39:39.3702539Z gym==0.26.2 2024-08-20T22:39:39.3702857Z gym-notices==0.0.8 2024-08-20T22:39:39.3703161Z h5py==3.11.0 2024-08-20T22:39:39.3703433Z higher==0.2.1 2024-08-20T22:39:39.3703767Z huggingface-hub==0.24.6 2024-08-20T22:39:39.3704137Z hydra-core==1.3.2 2024-08-20T22:39:39.3704436Z hypothesis==5.35.1 2024-08-20T22:39:39.3704751Z idna==3.7 2024-08-20T22:39:39.3705106Z imageio==2.35.1 2024-08-20T22:39:39.3705421Z importlib_metadata==8.4.0 2024-08-20T22:39:39.3705759Z inflect==7.3.1 2024-08-20T22:39:39.3706051Z iniconfig==2.0.0 2024-08-20T22:39:39.3706339Z iopath==0.1.9 2024-08-20T22:39:39.3706620Z isodate==0.6.1 2024-08-20T22:39:39.3706901Z isort==5.13.2 2024-08-20T22:39:39.3707183Z itsdangerous==2.2.0 2024-08-20T22:39:39.3707498Z Jinja2==3.1.4 2024-08-20T22:39:39.3707788Z jmespath==0.10.0 2024-08-20T22:39:39.3708086Z joblib==1.4.2 2024-08-20T22:39:39.3708363Z jsonpatch==1.33 2024-08-20T22:39:39.3708662Z jsonpointer==3.0.0 2024-08-20T22:39:39.3708970Z jsonschema==4.23.0 2024-08-20T22:39:39.3709359Z jsonschema-specifications==2023.12.1 2024-08-20T22:39:39.3709768Z junitparser==2.1.1 2024-08-20T22:39:39.3710108Z kaldi-io==0.9.8 2024-08-20T22:39:39.3710401Z kiwisolver==1.4.5 2024-08-20T22:39:39.3710692Z kornia==0.7.3 2024-08-20T22:39:39.3710975Z kornia_rs==0.1.5 2024-08-20T22:39:39.3711304Z lameenc==1.7.0 2024-08-20T22:39:39.3711889Z langcodes==3.4.0 2024-08-20T22:39:39.3712185Z langdetect==1.0.9 2024-08-20T22:39:39.3712672Z language-tags==1.2.0 2024-08-20T22:39:39.3712995Z language_data==1.2.0 2024-08-20T22:39:39.3713305Z lark==0.12.0 2024-08-20T22:39:39.3713594Z lazy_loader==0.4 2024-08-20T22:39:39.3713891Z libcst==1.4.0 2024-08-20T22:39:39.3714175Z librosa==0.9.2 2024-08-20T22:39:39.3714463Z lintrunner==0.12.5 2024-08-20T22:39:39.3714766Z llvmlite==0.38.1 2024-08-20T22:39:39.3715059Z lxml==4.9.4 2024-08-20T22:39:39.3715368Z marisa-trie==1.2.0 2024-08-20T22:39:39.3715677Z Markdown==3.7 2024-08-20T22:39:39.3716002Z markdown-it-py==3.0.0 2024-08-20T22:39:39.3716327Z MarkupSafe==2.1.5 2024-08-20T22:39:39.3716629Z matplotlib==3.8.4 2024-08-20T22:39:39.3716927Z mccabe==0.7.0 2024-08-20T22:39:39.3717212Z mdurl==0.1.2 2024-08-20T22:39:39.3717526Z ml-dtypes==0.4.0 2024-08-20T22:39:39.3717832Z MonkeyType==23.3.0 2024-08-20T22:39:39.3718182Z more-itertools==10.4.0 2024-08-20T22:39:39.3718504Z mpmath==1.3.0 2024-08-20T22:39:39.3718791Z msgpack==1.0.8 2024-08-20T22:39:39.3719080Z multidict==6.0.5 2024-08-20T22:39:39.3719388Z multiprocess==0.70.16 2024-08-20T22:39:39.3719704Z murmurhash==1.0.10 2024-08-20T22:39:39.3720005Z musdb==0.4.2 2024-08-20T22:39:39.3720287Z museval==0.4.1 2024-08-20T22:39:39.3720570Z mypy==1.10.0 2024-08-20T22:39:39.3720891Z mypy-extensions==1.0.0 2024-08-20T22:39:39.3721217Z networkx==2.8.8 2024-08-20T22:39:39.3721504Z ninja==1.10.2 2024-08-20T22:39:39.3721776Z nose==1.3.7 2024-08-20T22:39:39.3722046Z numba==0.55.2 2024-08-20T22:39:39.3722334Z numpy==1.21.2 2024-08-20T22:39:39.3722679Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:39:39.3723098Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:39:39.3723544Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:39:39.3723993Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:39:39.3724439Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:39:39.3724845Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:39:39.3725258Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:39:39.3725691Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:39:39.3726126Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:39:39.3726553Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:39:39.3726948Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:39:39.3727366Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:39:39.3727702Z oauthlib==3.2.2 2024-08-20T22:39:39.3728001Z omegaconf==2.3.0 2024-08-20T22:39:39.3728297Z onnx==1.16.1 2024-08-20T22:39:39.3728587Z onnxscript==0.1.0.dev20240817 2024-08-20T22:39:39.3728943Z opacus==1.5.2 2024-08-20T22:39:39.3729281Z opencv-python==4.8.0.74 2024-08-20T22:39:39.3729646Z opt-einsum==3.3.0 2024-08-20T22:39:39.3729938Z optree==0.12.1 2024-08-20T22:39:39.3730232Z packaging==24.1 2024-08-20T22:39:39.3730524Z pandas==2.0.3 2024-08-20T22:39:39.3730802Z patch==1.16 2024-08-20T22:39:39.3731091Z pathspec==0.12.1 2024-08-20T22:39:39.3731395Z phonemizer==3.3.0 2024-08-20T22:39:39.3731698Z pillow==10.3.0 2024-08-20T22:39:39.3731989Z platformdirs==4.2.2 2024-08-20T22:39:39.3732479Z pluggy==1.5.0 2024-08-20T22:39:39.3732771Z pooch==1.8.2 2024-08-20T22:39:39.3733060Z portalocker==2.10.1 2024-08-20T22:39:39.3733362Z preshed==3.0.9 2024-08-20T22:39:39.3733660Z prettytable==3.11.0 2024-08-20T22:39:39.3733974Z protobuf==3.20.2 2024-08-20T22:39:39.3734268Z psutil==6.0.0 2024-08-20T22:39:39.3734605Z py-cpuinfo==9.0.0 2024-08-20T22:39:39.3734908Z pyaml==24.7.0 2024-08-20T22:39:39.3735196Z pyarrow==17.0.0 2024-08-20T22:39:39.3735480Z pyasn1==0.6.0 2024-08-20T22:39:39.3735775Z pyasn1_modules==0.4.0 2024-08-20T22:39:39.3736103Z pyclipper==1.3.0.post5 2024-08-20T22:39:39.3736431Z pycocotools==2.0.8 2024-08-20T22:39:39.3736744Z pycparser==2.22 2024-08-20T22:39:39.3737051Z pycryptodomex==3.20.0 2024-08-20T22:39:39.3737371Z pydantic==2.8.2 2024-08-20T22:39:39.3737664Z pydantic_core==2.20.1 2024-08-20T22:39:39.3737985Z pydot==3.0.1 2024-08-20T22:39:39.3738276Z pygame==2.6.0 2024-08-20T22:39:39.3738558Z PyGithub==2.3.0 2024-08-20T22:39:39.3738855Z Pygments==2.15.0 2024-08-20T22:39:39.3739154Z PyJWT==2.9.0 2024-08-20T22:39:39.3739626Z pylatexenc==2.10 2024-08-20T22:39:39.3739920Z pylint==3.2.6 2024-08-20T22:39:39.3740203Z PyNaCl==1.5.0 2024-08-20T22:39:39.3740572Z pynvml==11.5.3 2024-08-20T22:39:39.3740861Z pyparsing==3.1.2 2024-08-20T22:39:39.3741165Z pypdfium2==4.30.0 2024-08-20T22:39:39.3741464Z pysbd==0.3.4 2024-08-20T22:39:39.3741747Z PySocks==1.7.1 2024-08-20T22:39:39.3742028Z pytest==7.3.2 2024-08-20T22:39:39.3742381Z pytest-benchmark==4.0.0 2024-08-20T22:39:39.3742751Z pytest-cpp==2.3.0 2024-08-20T22:39:39.3743112Z pytest-flakefinder==1.1.0 2024-08-20T22:39:39.3743514Z pytest-rerunfailures==14.0 2024-08-20T22:39:39.3743903Z pytest-xdist==3.3.1 2024-08-20T22:39:39.3744273Z python-dateutil==2.9.0.post0 2024-08-20T22:39:39.3745051Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:39:39.3746458Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:39:39.3747408Z pytz==2024.1 2024-08-20T22:39:39.3747709Z PyWavelets==1.4.1 2024-08-20T22:39:39.3748078Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:39:39.3748533Z rapidfuzz==3.9.6 2024-08-20T22:39:39.3748832Z rdflib==7.0.0 2024-08-20T22:39:39.3749115Z redis==5.0.8 2024-08-20T22:39:39.3749396Z referencing==0.35.1 2024-08-20T22:39:39.3749700Z regex==2024.7.24 2024-08-20T22:39:39.3750002Z requests==2.32.3 2024-08-20T22:39:39.3750350Z requests-oauthlib==2.0.0 2024-08-20T22:39:39.3750691Z resampy==0.4.3 2024-08-20T22:39:39.3750980Z rfc3986==1.5.0 2024-08-20T22:39:39.3751276Z rich==13.7.1 2024-08-20T22:39:39.3751587Z rockset==1.0.3 2024-08-20T22:39:39.3751907Z rpds-py==0.20.0 2024-08-20T22:39:39.3752194Z rsa==4.9 2024-08-20T22:39:39.3752457Z s3transfer==0.5.2 2024-08-20T22:39:39.3752762Z safetensors==0.4.4 2024-08-20T22:39:39.3753106Z scikit-image==0.21.0 2024-08-20T22:39:39.3753460Z scikit-learn==1.5.1 2024-08-20T22:39:39.3753761Z scipy==1.10.1 2024-08-20T22:39:39.3754052Z segments==2.2.1 2024-08-20T22:39:39.3754356Z sentencepiece==0.2.0 2024-08-20T22:39:39.3754705Z sentry-sdk==2.13.0 2024-08-20T22:39:39.3755015Z setproctitle==1.3.3 2024-08-20T22:39:39.3755330Z shapely==2.0.6 2024-08-20T22:39:39.3755624Z shellingham==1.5.4 2024-08-20T22:39:39.3755922Z simplejson==3.19.3 2024-08-20T22:39:39.3756321Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:39:39.3756822Z smart-open==7.0.4 2024-08-20T22:39:39.3757115Z smmap==5.0.1 2024-08-20T22:39:39.3757411Z sortedcontainers==2.4.0 2024-08-20T22:39:39.3757746Z soundfile==0.12.1 2024-08-20T22:39:39.3758045Z soupsieve==2.6 2024-08-20T22:39:39.3758329Z soxr==0.4.0 2024-08-20T22:39:39.3758605Z spacy==3.7.6 2024-08-20T22:39:39.3758930Z spacy-legacy==3.0.12 2024-08-20T22:39:39.3759286Z spacy-loggers==1.0.5 2024-08-20T22:39:39.3759601Z srsly==2.4.8 2024-08-20T22:39:39.3759881Z stempeg==0.2.3 2024-08-20T22:39:39.3760178Z submitit==1.5.1 2024-08-20T22:39:39.3760465Z sympy==1.13.1 2024-08-20T22:39:39.3760753Z tabulate==0.9.0 2024-08-20T22:39:39.3761094Z tb-nightly==2.13.0a20230426 2024-08-20T22:39:39.3761455Z tensorboard==2.13.0 2024-08-20T22:39:39.3761852Z tensorboard-data-server==0.7.2 2024-08-20T22:39:39.3762261Z tensorboardX==2.6.2.2 2024-08-20T22:39:39.3762589Z termcolor==2.4.0 2024-08-20T22:39:39.3762880Z thinc==8.2.5 2024-08-20T22:39:39.3763173Z threadpoolctl==3.5.0 2024-08-20T22:39:39.3763490Z tifffile==2024.8.10 2024-08-20T22:39:39.3764293Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:39:39.3765051Z tlparse==0.3.25 2024-08-20T22:39:39.3765344Z tokenizers==0.15.2 2024-08-20T22:39:39.3765645Z tomli==2.0.1 2024-08-20T22:39:39.3765923Z tomlkit==0.13.2 2024-08-20T22:39:39.3767111Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b444e33ace8c516878aca2862d707788a4c01d8c6190fb13e010fb9a131a57c5 2024-08-20T22:39:39.3768707Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:39:39.3769996Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:39:39.3771048Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:39:39.3772088Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:39:39.3773364Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:39:39.3774446Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:39:39.3775424Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:39:39.3776103Z tornado==6.4.1 2024-08-20T22:39:39.3776392Z tqdm==4.66.5 2024-08-20T22:39:39.3776680Z transformers==4.38.1 2024-08-20T22:39:39.3776998Z treetable==0.2.5 2024-08-20T22:39:39.3777799Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:39:39.3778579Z typeguard==4.3.0 2024-08-20T22:39:39.3778887Z typer==0.12.4 2024-08-20T22:39:39.3779187Z typing_extensions==4.12.2 2024-08-20T22:39:39.3779519Z tzdata==2024.1 2024-08-20T22:39:39.3779815Z Unidecode==1.3.8 2024-08-20T22:39:39.3780193Z unittest-xml-reporting==3.2.0 2024-08-20T22:39:39.3780562Z uritemplate==4.1.1 2024-08-20T22:39:39.3780866Z urllib3==1.26.19 2024-08-20T22:39:39.3781166Z visdom==0.2.4 2024-08-20T22:39:39.3781450Z wandb==0.17.7 2024-08-20T22:39:39.3781725Z wasabi==1.1.3 2024-08-20T22:39:39.3782014Z wcwidth==0.2.13 2024-08-20T22:39:39.3782306Z weasel==0.4.1 2024-08-20T22:39:39.3782639Z websocket-client==1.8.0 2024-08-20T22:39:39.3782973Z Werkzeug==3.0.3 2024-08-20T22:39:39.3783268Z wrapt==1.16.0 2024-08-20T22:39:39.3783554Z xdoctest==1.1.0 2024-08-20T22:39:39.3783838Z xxhash==3.5.0 2024-08-20T22:39:39.3784127Z xyzservices==2024.6.0 2024-08-20T22:39:39.3784448Z yacs==0.1.8 2024-08-20T22:39:39.3784716Z yarl==1.9.4 2024-08-20T22:39:39.3785034Z z3-solver==4.12.2.0 2024-08-20T22:39:39.3785338Z zipp==3.20.0 2024-08-20T22:39:39.3981443Z + popd 2024-08-20T22:39:39.3981793Z ~/workspace 2024-08-20T22:39:39.3982128Z + [[ aot_inductor_torchbench != *cpu* ]] 2024-08-20T22:39:39.3982579Z + install_torchrec_and_fbgemm 2024-08-20T22:39:39.3982944Z + local torchrec_commit 2024-08-20T22:39:39.3986242Z ++ get_pinned_commit torchrec 2024-08-20T22:39:39.3986667Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-08-20T22:39:39.4022775Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T22:39:39.4023288Z + local fbgemm_commit 2024-08-20T22:39:39.4026769Z ++ get_pinned_commit fbgemm 2024-08-20T22:39:39.4027177Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-08-20T22:39:39.4111743Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:39.4112408Z + pip_uninstall torchrec-nightly 2024-08-20T22:39:39.4112876Z + pip uninstall -y torchrec-nightly 2024-08-20T22:39:39.7865212Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-08-20T22:39:39.8237958Z + pip_uninstall fbgemm-gpu-nightly 2024-08-20T22:39:39.8238628Z + pip uninstall -y fbgemm-gpu-nightly 2024-08-20T22:39:40.1928656Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-08-20T22:39:40.2305852Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:40.2306845Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:40.7392744Z Collecting setuptools-git-versioning 2024-08-20T22:39:40.7587459Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-08-20T22:39:40.8584412Z Collecting scikit-build 2024-08-20T22:39:40.8610820Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-08-20T22:39:40.9777542Z Collecting pyre-extensions 2024-08-20T22:39:40.9803699Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-08-20T22:39:41.0336327Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-08-20T22:39:41.0340073Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0) 2024-08-20T22:39:41.0887603Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-08-20T22:39:41.0914950Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-08-20T22:39:41.1439613Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-08-20T22:39:41.1444500Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-08-20T22:39:41.1448455Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-08-20T22:39:41.1988491Z Collecting typing-inspect (from pyre-extensions) 2024-08-20T22:39:41.2014740Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-08-20T22:39:41.2517041Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-08-20T22:39:41.2595187Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-08-20T22:39:41.2629925Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-08-20T22:39:41.3113044Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-08-20T22:39:41.3673671Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-08-20T22:39:41.4202760Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-08-20T22:39:41.4690988Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-08-20T22:39:42.3541477Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-08-20T22:39:42.7066936Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-08-20T22:39:42.8193400Z + CUDA_PATH=/usr/local/cuda-12.1 2024-08-20T22:39:42.8194633Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:42.8196495Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:43.1905487Z Collecting fbgemm-gpu 2024-08-20T22:39:43.1908113Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-hqrrbix8/fbgemm-gpu_9bfa30d43f70456693ccfe1868c5711c 2024-08-20T22:39:43.1938714Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-hqrrbix8/fbgemm-gpu_9bfa30d43f70456693ccfe1868c5711c 2024-08-20T22:39:44.2657691Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-08-20T22:39:44.2683764Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:44.7620216Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:45.2775248Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:45.2776687Z Running command git submodule update --init --recursive -q 2024-08-20T22:39:54.4949433Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:39:54.4968701Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-08-20T22:39:54.4976680Z Building wheels for collected packages: fbgemm-gpu 2024-08-20T23:25:19.6194231Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T23:25:19.8026372Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252619849 sha256=c5a90dab8c63017928f0dd6165ad0c0167116a2f27ad8809f7786a16c00de6f1 2024-08-20T23:25:19.8028742Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-08-20T23:25:19.8048486Z Successfully built fbgemm-gpu 2024-08-20T23:25:20.6535990Z Installing collected packages: fbgemm-gpu 2024-08-20T23:25:24.0482382Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-08-20T23:25:24.9691026Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:24.9692707Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:25.3392079Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:25.3395714Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-ij29gyx7 2024-08-20T23:25:25.3427302Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-ij29gyx7 2024-08-20T23:25:26.1209781Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-08-20T23:25:26.1236743Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:26.4297291Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:26.7778843Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:27.0502629Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T23:25:27.0538185Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-08-20T23:25:27.0541381Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-08-20T23:25:27.0543770Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-08-20T23:25:27.1028849Z Collecting torchmetrics (from torchrec==0.3.2) 2024-08-20T23:25:27.1198647Z Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB) 2024-08-20T23:25:27.1611338Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5) 2024-08-20T23:25:27.1613083Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-08-20T23:25:27.1624113Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-08-20T23:25:27.1633900Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1) 2024-08-20T23:25:27.1896532Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-08-20T23:25:27.1900335Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:25:27.1903667Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:25:27.1926443Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-08-20T23:25:27.1930353Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6) 2024-08-20T23:25:27.2079915Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-08-20T23:25:27.2106589Z Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) 2024-08-20T23:25:27.2197685Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0) 2024-08-20T23:25:27.2202058Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-08-20T23:25:27.2223057Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-08-20T23:25:27.2258069Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-08-20T23:25:27.2262148Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-08-20T23:25:27.2264616Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-08-20T23:25:27.2268121Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1) 2024-08-20T23:25:27.2273343Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1) 2024-08-20T23:25:27.2289153Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-08-20T23:25:27.2701809Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-08-20T23:25:27.2840694Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB) 2024-08-20T23:25:27.3041594Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB) 2024-08-20T23:25:27.3145693Z Building wheels for collected packages: torchrec 2024-08-20T23:25:27.8471898Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-08-20T23:25:27.8481585Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=91287a39eba77997adc15b17abfccb427eb07fecf6fb397136d79194e86e736f 2024-08-20T23:25:27.8483454Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-08-20T23:25:27.8505693Z Successfully built torchrec 2024-08-20T23:25:28.6386080Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-08-20T23:25:29.2538332Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2 2024-08-20T23:25:29.3951965Z ++ pwd 2024-08-20T23:25:29.3954385Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T23:25:29.3955002Z + test_dynamo_benchmark torchbench 0 2024-08-20T23:25:29.3960596Z ++ pwd 2024-08-20T23:25:29.3964266Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:29.3964823Z + local suite=torchbench 2024-08-20T23:25:29.3966999Z + shift 2024-08-20T23:25:29.3967569Z + local shard_id=0 2024-08-20T23:25:29.3968116Z + shift 2024-08-20T23:25:29.3968561Z + [[ aot_inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:29.3969084Z + [[ aot_inductor_torchbench == *perf* ]] 2024-08-20T23:25:29.3969655Z + [[ aot_inductor_torchbench == *cpu* ]] 2024-08-20T23:25:29.3970152Z + [[ aot_inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:29.3970939Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-08-20T23:25:29.3971561Z ++ pwd 2024-08-20T23:25:29.3972240Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:29.3972931Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:29.4054575Z + local name=inference 2024-08-20T23:25:29.4055109Z + shift 2024-08-20T23:25:29.4055440Z + local suite=torchbench 2024-08-20T23:25:29.4055779Z + shift 2024-08-20T23:25:29.4056046Z + local shard_id=0 2024-08-20T23:25:29.4056350Z + shift 2024-08-20T23:25:29.4056624Z + partition_flags=() 2024-08-20T23:25:29.4056959Z + local partition_flags 2024-08-20T23:25:29.4057361Z + [[ -n 2 ]] 2024-08-20T23:25:29.4057680Z + [[ -n 0 ]] 2024-08-20T23:25:29.4061635Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:25:29.4062503Z + [[ aot_inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:29.4062998Z + [[ aot_inductor_torchbench == *perf* ]] 2024-08-20T23:25:29.4063477Z + [[ aot_inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:29.4064034Z + [[ aot_inductor_torchbench != *cpu_aot_inductor* ]] 2024-08-20T23:25:29.4064525Z + export TORCHINDUCTOR_ABI_COMPATIBLE=1 2024-08-20T23:25:29.4064965Z + TORCHINDUCTOR_ABI_COMPATIBLE=1 2024-08-20T23:25:29.4065373Z + [[ aot_inductor_torchbench == *_avx2* ]] 2024-08-20T23:25:29.4065831Z + [[ aot_inductor_torchbench == *_avx512* ]] 2024-08-20T23:25:29.4067491Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --export-aot-inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T23:25:35.4021964Z 2024-08-20T23:25:36.1869042Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-08-20T23:25:36.1871041Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-08-20T23:25:36.6151749Z 2024-08-20T23:25:36.6152453Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:25:36.6152982Z cuda eval torchrec_dlrm 2024-08-20T23:25:36.6178857Z Traceback (most recent call last): 2024-08-20T23:25:36.6179882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:25:36.6180800Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:25:36.6181915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:25:36.6182999Z return mod(*inputs) 2024-08-20T23:25:36.6184325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:36.6185233Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:36.6186181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:36.6187023Z return forward_call(*args, **kwargs) 2024-08-20T23:25:36.6187897Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-08-20T23:25:36.6188809Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-08-20T23:25:36.6189915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:36.6190813Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:36.6191744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:36.6192577Z return forward_call(*args, **kwargs) 2024-08-20T23:25:36.6193445Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-08-20T23:25:36.6194235Z concatenated_dense = self.inter_arch( 2024-08-20T23:25:36.6195213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:36.6196091Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:36.6197025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:36.6198201Z return forward_call(*args, **kwargs) 2024-08-20T23:25:36.6199085Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-08-20T23:25:36.6200011Z return self.crossnet(combined_values.reshape([B, -1])) 2024-08-20T23:25:36.6201075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:36.6201962Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:36.6202899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:36.6203715Z return forward_call(*args, **kwargs) 2024-08-20T23:25:36.6204616Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-08-20T23:25:36.6205535Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-08-20T23:25:36.6206370Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-08-20T23:25:36.6206916Z 2024-08-20T23:25:36.6207214Z The above exception was the direct cause of the following exception: 2024-08-20T23:25:36.6207658Z 2024-08-20T23:25:36.6207803Z Traceback (most recent call last): 2024-08-20T23:25:36.6208460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:25:36.6209122Z ) = runner.load_model( 2024-08-20T23:25:36.6209785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:25:36.6210532Z self.validate_model(model, example_inputs) 2024-08-20T23:25:36.6211282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:25:36.6212117Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:25:36.6212656Z RuntimeError: Eager run failed 2024-08-20T23:25:36.6212978Z 2024-08-20T23:25:36.6213124Z eager_fail_to_run 2024-08-20T23:25:40.3355814Z 2024-08-20T23:25:42.5712115Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:42.5712594Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:25:42.5771210Z cuda eval BERT_pytorch 2024-08-20T23:25:43.6853799Z ERROR:common: 2024-08-20T23:25:43.6854273Z Traceback (most recent call last): 2024-08-20T23:25:43.6855731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T23:25:43.6856732Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T23:25:43.6857805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T23:25:43.6859050Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T23:25:43.6859900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T23:25:43.6860611Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T23:25:43.6861802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T23:25:43.6862663Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T23:25:43.6863631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T23:25:43.6864711Z result_traced = opt_f(*args, **kwargs) 2024-08-20T23:25:43.6866015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:43.6867131Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:43.6868373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:43.6869465Z return forward_call(*args, **kwargs) 2024-08-20T23:25:43.6870444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T23:25:43.6871198Z return fn(*args, **kwargs) 2024-08-20T23:25:43.6872190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:43.6873077Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:43.6874011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:43.6874834Z return forward_call(*args, **kwargs) 2024-08-20T23:25:43.6875760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T23:25:43.6876590Z return self._torchdynamo_orig_callable( 2024-08-20T23:25:43.6877521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T23:25:43.6878305Z return _compile( 2024-08-20T23:25:43.6879115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T23:25:43.6880042Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T23:25:43.6881129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T23:25:43.6882052Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T23:25:43.6883057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T23:25:43.6883869Z return function(*args, **kwargs) 2024-08-20T23:25:43.6884799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T23:25:43.6885692Z out_code = transform_code_object(code, transform) 2024-08-20T23:25:43.6886796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T23:25:43.6887775Z transformations(instructions, code_options) 2024-08-20T23:25:43.6888684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T23:25:43.6889460Z return fn(*args, **kwargs) 2024-08-20T23:25:43.6890331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T23:25:43.6891251Z tracer.run() 2024-08-20T23:25:43.6892287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T23:25:43.6893079Z super().run() 2024-08-20T23:25:43.6893880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:25:43.6894660Z while self.step(): 2024-08-20T23:25:43.6895473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:25:43.6896324Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:25:43.6897278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T23:25:43.6898382Z return inner_fn(self, inst) 2024-08-20T23:25:43.6899313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T23:25:43.6900180Z self.call_function(fn, args, {}) 2024-08-20T23:25:43.6901128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T23:25:43.6902222Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T23:25:43.6903361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T23:25:43.6904277Z return super().call_function(tx, args, kwargs) 2024-08-20T23:25:43.6905298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T23:25:43.6906196Z return super().call_function(tx, args, kwargs) 2024-08-20T23:25:43.6907209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T23:25:43.6908258Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T23:25:43.6909485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T23:25:43.6910592Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T23:25:43.6911726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T23:25:43.6912630Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T23:25:43.6913648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T23:25:43.6914468Z tracer.run() 2024-08-20T23:25:43.6915259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:25:43.6916033Z while self.step(): 2024-08-20T23:25:43.6916856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:25:43.6917708Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:25:43.6918667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T23:25:43.6919478Z return inner_fn(self, inst) 2024-08-20T23:25:43.6920391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T23:25:43.6921247Z self.call_function(fn, args, {}) 2024-08-20T23:25:43.6922193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T23:25:43.6923279Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T23:25:43.6924416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T23:25:43.6925325Z return super().call_function(tx, args, kwargs) 2024-08-20T23:25:43.6927032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T23:25:43.6927952Z return super().call_function(tx, args, kwargs) 2024-08-20T23:25:43.6928958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T23:25:43.6930011Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T23:25:43.6931231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T23:25:43.6932407Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T23:25:43.6933542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T23:25:43.6934446Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T23:25:43.6935489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T23:25:43.6936311Z tracer.run() 2024-08-20T23:25:43.6937109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:25:43.6937882Z while self.step(): 2024-08-20T23:25:43.6938699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:25:43.6939686Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:25:43.6940900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1739, in STORE_ATTR 2024-08-20T23:25:43.6941705Z assert ( 2024-08-20T23:25:43.6942136Z AssertionError: Mutating module attribute mask during export. 2024-08-20T23:25:43.6942546Z 2024-08-20T23:25:43.6942666Z from user code: 2024-08-20T23:25:43.6943517Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 50, in forward 2024-08-20T23:25:43.6944448Z x = transformer.forward(x, mask) 2024-08-20T23:25:43.6945422Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 47, in forward 2024-08-20T23:25:43.6946398Z self.lambda_module.set_mask(mask) 2024-08-20T23:25:43.6947373Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 16, in set_mask 2024-08-20T23:25:43.6948327Z self.mask = mask 2024-08-20T23:25:43.6948531Z 2024-08-20T23:25:43.6948827Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T23:25:43.6949276Z 2024-08-20T23:25:43.6949570Z TorchDynamo optimized model failed to run because of following error 2024-08-20T23:25:43.6950123Z fail_to_run 2024-08-20T23:25:43.6950402Z TIMING: 2024-08-20T23:25:43.6950958Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:12 | FakeTensorMode.__torch_dispatch__:43 2024-08-20T23:25:43.6951834Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:25:47.2905891Z 2024-08-20T23:25:50.3261419Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:50.3261889Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:25:50.3309282Z cuda eval Background_Matting 2024-08-20T23:25:50.3312429Z pass_due_to_skip 2024-08-20T23:25:50.3852340Z TIMING: 2024-08-20T23:25:50.3852642Z STATS: call_* op count: 0 2024-08-20T23:25:50.3853188Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:25:54.0151225Z 2024-08-20T23:25:57.4228371Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:57.4229121Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:25:57.4255025Z cuda eval LearningToPaint 2024-08-20T23:26:18.4710184Z pass 2024-08-20T23:26:18.4711342Z TIMING: backend_compile:9e-05 entire_frame_compile:0.61857 total_wall_time:0.61857 2024-08-20T23:26:18.4713586Z STATS: call_* op count: 71 | FakeTensor.__torch_dispatch__:1188 | FakeTensorMode.__torch_dispatch__:6727 | ProxyTorchDispatchMode.__torch_dispatch__:1808 2024-08-20T23:26:18.4716783Z Dynamo produced 1 graphs covering 71 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:22.7270729Z 2024-08-20T23:26:24.1705891Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-08-20T23:26:24.1866872Z 2024-08-20T23:26:24.1866881Z 2024-08-20T23:26:24.2872413Z 0% 0.00/528M [00:00 2024-08-20T23:28:48.2590459Z _to_bytes(graph.get_original_value_of_constant(name), all_cuda) 2024-08-20T23:28:48.2591561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1801, in _to_bytes 2024-08-20T23:28:48.2592385Z t_cpu = t.untyped_storage().cpu() 2024-08-20T23:28:48.2593231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/storage.py", line 252, in cpu 2024-08-20T23:28:48.2594054Z return torch.UntypedStorage(self.size()).copy_(self, False) 2024-08-20T23:28:48.2594730Z RuntimeError: CUDA error: device-side assert triggered 2024-08-20T23:28:48.2595641Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 2024-08-20T23:28:48.2596543Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1 2024-08-20T23:28:48.2597249Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 2024-08-20T23:28:48.2598085Z 2024-08-20T23:28:48.2598092Z 2024-08-20T23:28:48.2598486Z During handling of the above exception, another exception occurred: 2024-08-20T23:28:48.2599124Z 2024-08-20T23:28:48.2599326Z Traceback (most recent call last): 2024-08-20T23:28:48.2600388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T23:28:48.2601512Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T23:28:48.2602660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T23:28:48.2604176Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T23:28:48.2605488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1376, in load 2024-08-20T23:28:48.2606580Z so_path = torch._inductor.aot_compile( 2024-08-20T23:28:48.2608070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/__init__.py", line 115, in aot_compile 2024-08-20T23:28:48.2609329Z return compile_fx_aot( 2024-08-20T23:28:48.2610717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1141, in compile_fx_aot 2024-08-20T23:28:48.2612055Z compiled_lib_path = compile_fx( 2024-08-20T23:28:48.2613618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1251, in compile_fx 2024-08-20T23:28:48.2614892Z return compile_fx( 2024-08-20T23:28:48.2616230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1293, in compile_fx 2024-08-20T23:28:48.2617424Z return compile_fx( 2024-08-20T23:28:48.2618759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1512, in compile_fx 2024-08-20T23:28:48.2620205Z return inference_compiler(unlifted_gm, example_inputs_) 2024-08-20T23:28:48.2621862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1348, in fw_compiler_base 2024-08-20T23:28:48.2623442Z return _fw_compiler_base(model, example_inputs, is_inference) 2024-08-20T23:28:48.2625078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1419, in _fw_compiler_base 2024-08-20T23:28:48.2626833Z return inner_compile( 2024-08-20T23:28:48.2627722Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-08-20T23:28:48.2628702Z return func(*args, **kwds) 2024-08-20T23:28:48.2630162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 473, in compile_fx_inner 2024-08-20T23:28:48.2631713Z return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")( 2024-08-20T23:28:48.2633203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 85, in debug_wrapper 2024-08-20T23:28:48.2634250Z inner_compiled_fn = compiler_fn(gm, example_inputs) 2024-08-20T23:28:48.2635321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 669, in _compile_fx_inner 2024-08-20T23:28:48.2636182Z compiled_graph = codegen_and_compile( 2024-08-20T23:28:48.2637174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 568, in codegen_and_compile 2024-08-20T23:28:48.2638189Z compiled_graph = fx_codegen_and_compile(gm, example_inputs, **fx_kwargs) 2024-08-20T23:28:48.2639330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 716, in fx_codegen_and_compile 2024-08-20T23:28:48.2640225Z with dynamo_utils.preserve_rng_state(): 2024-08-20T23:28:48.2640900Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 153, in __exit__ 2024-08-20T23:28:48.2641571Z self.gen.throw(typ, value, traceback) 2024-08-20T23:28:48.2642509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1078, in preserve_rng_state 2024-08-20T23:28:48.2643588Z torch.cuda.set_rng_state(cuda_rng_state) # type: ignore[possibly-undefined] 2024-08-20T23:28:48.2644660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/random.py", line 77, in set_rng_state 2024-08-20T23:28:48.2645424Z _lazy_call(cb) 2024-08-20T23:28:48.2646200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py", line 245, in _lazy_call 2024-08-20T23:28:48.2646958Z callable() 2024-08-20T23:28:48.2647673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/random.py", line 75, in cb 2024-08-20T23:28:48.2648439Z default_generator.set_state(new_state_copy) 2024-08-20T23:28:48.2649036Z RuntimeError: CUDA error: device-side assert triggered 2024-08-20T23:28:48.2649946Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 2024-08-20T23:28:48.2650840Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1 2024-08-20T23:28:48.2651540Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 2024-08-20T23:28:48.2651972Z 2024-08-20T23:28:48.2652397Z TorchDynamo optimized model failed to run because of following error 2024-08-20T23:28:48.2652962Z fail_to_run 2024-08-20T23:28:48.2653263Z Traceback (most recent call last): 2024-08-20T23:28:48.2654209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in 2024-08-20T23:28:48.2655196Z torchbench_main() 2024-08-20T23:28:48.2656048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torchbench_main 2024-08-20T23:28:48.2656907Z main(TorchBenchmarkRunner(), original_dir) 2024-08-20T23:28:48.2657687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4087, in main 2024-08-20T23:28:48.2658403Z process_entry(0, runner, original_dir, args) 2024-08-20T23:28:48.2659154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4019, in process_entry 2024-08-20T23:28:48.2659878Z return run(runner, args, original_dir) 2024-08-20T23:28:48.2660557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4689, in run 2024-08-20T23:28:48.2661203Z runner.run_one_model( 2024-08-20T23:28:48.2662063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3383, in run_one_model 2024-08-20T23:28:48.2662795Z empty_gpu_cache(current_device) 2024-08-20T23:28:48.2663529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 416, in empty_gpu_cache 2024-08-20T23:28:48.2664242Z torch.cuda.empty_cache() 2024-08-20T23:28:48.2665105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/memory.py", line 179, in empty_cache 2024-08-20T23:28:48.2665888Z torch._C._cuda_emptyCache() 2024-08-20T23:28:48.2666423Z RuntimeError: CUDA error: device-side assert triggered 2024-08-20T23:28:48.2667336Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 2024-08-20T23:28:48.2668240Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1 2024-08-20T23:28:48.2668933Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 2024-08-20T23:28:48.2669372Z 2024-08-20T23:28:49.5575609Z Run failed with return code: 1 2024-08-20T23:28:49.5576131Z Output: None 2024-08-20T23:28:49.5576462Z Error: None 2024-08-20T23:28:52.3991223Z 2024-08-20T23:28:54.3228070Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:28:54.3228569Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:28:54.3232795Z cuda eval basic_gnn_gin 2024-08-20T23:29:05.9994662Z pass 2024-08-20T23:29:06.0089594Z TIMING: backend_compile:9e-05 entire_frame_compile:0.39339 total_wall_time:0.39339 2024-08-20T23:29:06.0090802Z STATS: call_* op count: 55 | FakeTensorMode.__torch_dispatch__:1521 | ProxyTorchDispatchMode.__torch_dispatch__:474 | FakeTensor.__torch_dispatch__:103 2024-08-20T23:29:06.0091908Z Dynamo produced 1 graphs covering 55 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:09.9005029Z 2024-08-20T23:29:11.8169018Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:11.8169566Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:29:11.8170740Z cuda eval basic_gnn_sage 2024-08-20T23:29:23.5468344Z pass 2024-08-20T23:29:23.5563914Z TIMING: backend_compile:0.0001 entire_frame_compile:0.33283 total_wall_time:0.33283 2024-08-20T23:29:23.5565211Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:1516 | ProxyTorchDispatchMode.__torch_dispatch__:466 | FakeTensor.__torch_dispatch__:138 2024-08-20T23:29:23.5566311Z Dynamo produced 1 graphs covering 58 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:27.4594787Z 2024-08-20T23:29:27.9694276Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:27.9694840Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:27.9697946Z cuda eval dcgan 2024-08-20T23:29:41.0124427Z pass 2024-08-20T23:29:41.0125769Z TIMING: backend_compile:8e-05 entire_frame_compile:0.16137 total_wall_time:0.16137 2024-08-20T23:29:41.0127494Z STATS: call_* op count: 13 | FakeTensor.__torch_dispatch__:220 | FakeTensorMode.__torch_dispatch__:1321 | ProxyTorchDispatchMode.__torch_dispatch__:320 2024-08-20T23:29:41.0129137Z Dynamo produced 1 graphs covering 13 ops with 0 graph breaks (0 unique) 2024-08-20T23:29:44.8538369Z 2024-08-20T23:29:47.7456641Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:47.7457129Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:29:47.7476158Z cuda eval demucs 2024-08-20T23:33:21.7505286Z pass 2024-08-20T23:33:21.7603182Z TIMING: backend_compile:0.00021 entire_frame_compile:3.23689 total_wall_time:3.23689 2024-08-20T23:33:21.7605952Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:132815 | FakeTensor.__torch_dispatch__:15527 | ProxyTorchDispatchMode.__torch_dispatch__:38531 2024-08-20T23:33:21.7607097Z Dynamo produced 1 graphs covering 67 ops with 0 graph breaks (0 unique) 2024-08-20T23:33:28.5889278Z 2024-08-20T23:33:28.8568294Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/densenet121-a639ec97.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/densenet121-a639ec97.pth 2024-08-20T23:33:28.8747240Z 2024-08-20T23:33:28.8747399Z 2024-08-20T23:33:28.9662346Z 0% 0.00/30.8M [00:00 2024-08-20T23:37:05.5709615Z [ 2024-08-20T23:37:05.5710402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/detection/core.py", line 92, in 2024-08-20T23:37:05.5711380Z cv2.morphologyEx(bmap[..., idx], cv2.MORPH_OPEN, self._opening_kernel) 2024-08-20T23:37:05.5711824Z 2024-08-20T23:37:05.5712126Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T23:37:05.5712568Z 2024-08-20T23:37:05.5712871Z TorchDynamo optimized model failed to run because of following error 2024-08-20T23:37:05.5713415Z fail_to_run 2024-08-20T23:37:05.6023445Z TIMING: 2024-08-20T23:37:05.6024141Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:605 | FakeTensorMode.__torch_dispatch__:3633 2024-08-20T23:37:05.6025014Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:37:09.3672385Z 2024-08-20T23:37:12.5683039Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead 2024-08-20T23:37:13.0150804Z 2024-08-20T23:37:13.0151636Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:37:13.0152704Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead 2024-08-20T23:37:13.0153446Z cuda eval doctr_reco_predictor 2024-08-20T23:37:13.1788754Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead 2024-08-20T23:37:14.9401754Z W0820 23:37:14.939000 38534 torch/fx/experimental/symbolic_shapes.py:5131] [0/0] failed during evaluate_expr(Ne(u0, 123), hint=None, expect_rational=True, size_oblivious=False, forcing_spec=False 2024-08-20T23:37:14.9403891Z E0820 23:37:14.939000 38534 torch/fx/experimental/recording.py:298] [0/0] failed while running evaluate_expr(*(Ne(u0, 123), None), **{'fx_node': ne}) 2024-08-20T23:37:14.9482133Z ERROR:common: 2024-08-20T23:37:14.9482927Z Traceback (most recent call last): 2024-08-20T23:37:14.9484236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T23:37:14.9485145Z return guard_scalar(self.sym_num) 2024-08-20T23:37:14.9486176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 845, in guard_scalar 2024-08-20T23:37:14.9487155Z return guard_bool(a) 2024-08-20T23:37:14.9488429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 1018, in guard_bool 2024-08-20T23:37:14.9489757Z return a.node.guard_bool("", 0) # NB: uses Python backtrace 2024-08-20T23:37:14.9491161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/sym_node.py", line 451, in guard_bool 2024-08-20T23:37:14.9492335Z r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node) 2024-08-20T23:37:14.9493485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/recording.py", line 292, in wrapper 2024-08-20T23:37:14.9494420Z return retlog(event.run(self)) 2024-08-20T23:37:14.9495328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/recording.py", line 165, in run 2024-08-20T23:37:14.9496149Z return self.f(*args, **kwargs) 2024-08-20T23:37:14.9497148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5129, in evaluate_expr 2024-08-20T23:37:14.9498776Z return self._evaluate_expr(orig_expr, hint, fx_node, expect_rational, size_oblivious, forcing_spec=forcing_spec) 2024-08-20T23:37:14.9500166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5168, in _evaluate_expr 2024-08-20T23:37:14.9501113Z concrete_val = compute_concrete_val() 2024-08-20T23:37:14.9502202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 5149, in compute_concrete_val 2024-08-20T23:37:14.9503221Z return self.size_hint(orig_expr) 2024-08-20T23:37:14.9504223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4671, in size_hint 2024-08-20T23:37:14.9505236Z raise self._make_data_dependent_error(result_expr, expr) 2024-08-20T23:37:14.9506594Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not guard on data-dependent expression Ne(u0, 123) (unhinted: Ne(u0, 123)). (Size-like symbols: none) 2024-08-20T23:37:14.9507573Z 2024-08-20T23:37:14.9507854Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T23:37:14.9508965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T23:37:14.9509836Z return guard_scalar(self.sym_num) 2024-08-20T23:37:14.9510128Z 2024-08-20T23:37:14.9510324Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T23:37:14.9511111Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T23:37:14.9512354Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T23:37:14.9513734Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T23:37:14.9514527Z 2024-08-20T23:37:14.9514677Z User Stack (most recent call last): 2024-08-20T23:37:14.9515114Z (snipped, see stack below for prefix) 2024-08-20T23:37:14.9516129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T23:37:14.9517059Z out["preds"] = self.postprocessor(logits) 2024-08-20T23:37:14.9518079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T23:37:14.9519183Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T23:37:14.9520424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T23:37:14.9521328Z words = [ 2024-08-20T23:37:14.9522221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T23:37:14.9523303Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T23:37:14.9524684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T23:37:14.9526090Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T23:37:14.9526556Z 2024-08-20T23:37:14.9526819Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T23:37:14.9527224Z 2024-08-20T23:37:14.9527520Z During handling of the above exception, another exception occurred: 2024-08-20T23:37:14.9527969Z 2024-08-20T23:37:14.9528118Z Traceback (most recent call last): 2024-08-20T23:37:14.9528828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T23:37:14.9529591Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T23:37:14.9530369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T23:37:14.9531294Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T23:37:14.9532254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T23:37:14.9533022Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T23:37:14.9534027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T23:37:14.9534920Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T23:37:14.9535839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1437, in inner 2024-08-20T23:37:14.9536652Z result_traced = opt_f(*args, **kwargs) 2024-08-20T23:37:14.9537635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:37:14.9538543Z return self._call_impl(*args, **kwargs) 2024-08-20T23:37:14.9539486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:37:14.9540323Z return forward_call(*args, **kwargs) 2024-08-20T23:37:14.9541198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-20T23:37:14.9541966Z return fn(*args, **kwargs) 2024-08-20T23:37:14.9542906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:37:14.9543802Z return self._call_impl(*args, **kwargs) 2024-08-20T23:37:14.9544741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:37:14.9545572Z return forward_call(*args, **kwargs) 2024-08-20T23:37:14.9546701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-20T23:37:14.9547558Z return self._torchdynamo_orig_callable( 2024-08-20T23:37:14.9548489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-20T23:37:14.9549281Z return _compile( 2024-08-20T23:37:14.9550110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-20T23:37:14.9551052Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-20T23:37:14.9552190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-20T23:37:14.9553154Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-20T23:37:14.9554175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-20T23:37:14.9555195Z return function(*args, **kwargs) 2024-08-20T23:37:14.9556426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 686, in _compile_inner 2024-08-20T23:37:14.9557450Z out_code = transform_code_object(code, transform) 2024-08-20T23:37:14.9558573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object 2024-08-20T23:37:14.9559560Z transformations(instructions, code_options) 2024-08-20T23:37:14.9560484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 208, in _fn 2024-08-20T23:37:14.9561274Z return fn(*args, **kwargs) 2024-08-20T23:37:14.9562148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 622, in transform 2024-08-20T23:37:14.9562952Z tracer.run() 2024-08-20T23:37:14.9563758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2780, in run 2024-08-20T23:37:14.9564564Z super().run() 2024-08-20T23:37:14.9565362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:37:14.9566158Z while self.step(): 2024-08-20T23:37:14.9566989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:37:14.9567855Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:37:14.9568814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T23:37:14.9569645Z return inner_fn(self, inst) 2024-08-20T23:37:14.9570579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T23:37:14.9571444Z self.call_function(fn, args, {}) 2024-08-20T23:37:14.9572521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T23:37:14.9573659Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T23:37:14.9574852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 876, in call_function 2024-08-20T23:37:14.9575831Z return self.call_method(tx, "__call__", args, kwargs) 2024-08-20T23:37:14.9576893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 736, in call_method 2024-08-20T23:37:14.9577929Z return UserMethodVariable(method, self, source=source).call_function( 2024-08-20T23:37:14.9579078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 383, in call_function 2024-08-20T23:37:14.9580000Z return super().call_function(tx, args, kwargs) 2024-08-20T23:37:14.9581026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T23:37:14.9582209Z return super().call_function(tx, args, kwargs) 2024-08-20T23:37:14.9583247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T23:37:14.9584316Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T23:37:14.9585548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T23:37:14.9586667Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T23:37:14.9587812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T23:37:14.9588739Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T23:37:14.9589770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T23:37:14.9590622Z tracer.run() 2024-08-20T23:37:14.9591429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:37:14.9592222Z while self.step(): 2024-08-20T23:37:14.9593050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:37:14.9593908Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:37:14.9594867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T23:37:14.9595694Z return inner_fn(self, inst) 2024-08-20T23:37:14.9596649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1676, in CALL_FUNCTION_KW 2024-08-20T23:37:14.9597898Z self.call_function(fn, args, kwargs) 2024-08-20T23:37:14.9598879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T23:37:14.9599996Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T23:37:14.9601155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 322, in call_function 2024-08-20T23:37:14.9602080Z return super().call_function(tx, args, kwargs) 2024-08-20T23:37:14.9603110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T23:37:14.9604169Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T23:37:14.9605400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T23:37:14.9606518Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T23:37:14.9607668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T23:37:14.9608595Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T23:37:14.9609623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T23:37:14.9610460Z tracer.run() 2024-08-20T23:37:14.9611257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:37:14.9612052Z while self.step(): 2024-08-20T23:37:14.9612998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:37:14.9613858Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:37:14.9614821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T23:37:14.9615641Z return inner_fn(self, inst) 2024-08-20T23:37:14.9616904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T23:37:14.9617775Z self.call_function(fn, args, {}) 2024-08-20T23:37:14.9618727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T23:37:14.9619825Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T23:37:14.9620973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T23:37:14.9622043Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T23:37:14.9623276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T23:37:14.9624394Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T23:37:14.9625554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T23:37:14.9626474Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T23:37:14.9627504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T23:37:14.9628341Z tracer.run() 2024-08-20T23:37:14.9629154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:37:14.9629941Z while self.step(): 2024-08-20T23:37:14.9630773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:37:14.9631629Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:37:14.9632596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 578, in wrapper 2024-08-20T23:37:14.9633417Z return inner_fn(self, inst) 2024-08-20T23:37:14.9634362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1586, in CALL_FUNCTION 2024-08-20T23:37:14.9635226Z self.call_function(fn, args, {}) 2024-08-20T23:37:14.9636181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 826, in call_function 2024-08-20T23:37:14.9637274Z self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type] 2024-08-20T23:37:14.9638434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 106, in call_function 2024-08-20T23:37:14.9639502Z return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs) 2024-08-20T23:37:14.9640733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 832, in inline_user_function_return 2024-08-20T23:37:14.9641846Z return InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 2024-08-20T23:37:14.9643107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2995, in inline_call 2024-08-20T23:37:14.9644044Z return cls.inline_call_(parent, func, args, kwargs) 2024-08-20T23:37:14.9645085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3123, in inline_call_ 2024-08-20T23:37:14.9645925Z tracer.run() 2024-08-20T23:37:14.9646716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 979, in run 2024-08-20T23:37:14.9647512Z while self.step(): 2024-08-20T23:37:14.9648346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 891, in step 2024-08-20T23:37:14.9649207Z self.dispatch_table[inst.opcode](self, inst) 2024-08-20T23:37:14.9650155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 532, in inner 2024-08-20T23:37:14.9651155Z eval_result = value.evaluate_expr(self.output) 2024-08-20T23:37:14.9652386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1105, in evaluate_expr 2024-08-20T23:37:14.9653269Z raise UserError( # noqa: B904 2024-08-20T23:37:14.9654559Z torch._dynamo.exc.UserError: Consider annotating your code using torch._check*(). Could not guard on data-dependent expression Ne(u0, 123) (unhinted: Ne(u0, 123)). (Size-like symbols: none) 2024-08-20T23:37:14.9655574Z 2024-08-20T23:37:14.9655853Z Potential framework code culprit (scroll up for full backtrace): 2024-08-20T23:37:14.9656962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 1103, in evaluate_expr 2024-08-20T23:37:14.9657840Z return guard_scalar(self.sym_num) 2024-08-20T23:37:14.9658124Z 2024-08-20T23:37:14.9658333Z For more information, run with TORCH_LOGS="dynamic" 2024-08-20T23:37:14.9659110Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0" 2024-08-20T23:37:14.9660080Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T23:37:14.9661263Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing 2024-08-20T23:37:14.9662047Z 2024-08-20T23:37:14.9662202Z User Stack (most recent call last): 2024-08-20T23:37:14.9662687Z (snipped, see stack below for prefix) 2024-08-20T23:37:14.9663688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T23:37:14.9664619Z out["preds"] = self.postprocessor(logits) 2024-08-20T23:37:14.9665641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T23:37:14.9666733Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T23:37:14.9667973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T23:37:14.9668866Z words = [ 2024-08-20T23:37:14.9669755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T23:37:14.9670839Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T23:37:14.9672009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T23:37:14.9673077Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T23:37:14.9673545Z 2024-08-20T23:37:14.9673804Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1 2024-08-20T23:37:14.9674942Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example 2024-08-20T23:37:14.9675722Z 2024-08-20T23:37:14.9675846Z from user code: 2024-08-20T23:37:14.9676737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward 2024-08-20T23:37:14.9677662Z out["preds"] = self.postprocessor(logits) 2024-08-20T23:37:14.9678689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__ 2024-08-20T23:37:14.9679793Z return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab)) 2024-08-20T23:37:14.9681025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path 2024-08-20T23:37:14.9681919Z words = [ 2024-08-20T23:37:14.9682811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T23:37:14.9683886Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T23:37:14.9685255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in 2024-08-20T23:37:14.9686319Z decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab) 2024-08-20T23:37:14.9686794Z 2024-08-20T23:37:14.9687090Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-08-20T23:37:14.9687544Z 2024-08-20T23:37:14.9687840Z TorchDynamo optimized model failed to run because of following error 2024-08-20T23:37:14.9688433Z fail_to_run 2024-08-20T23:37:14.9688698Z TIMING: 2024-08-20T23:37:14.9689277Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:260 | FakeTensorMode.__torch_dispatch__:5942 2024-08-20T23:37:14.9690153Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:37:18.7615167Z 2024-08-20T23:37:20.5179416Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:20.5180147Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:37:20.5186131Z cuda eval drq 2024-08-20T23:37:20.9887884Z ERROR:common: 2024-08-20T23:37:20.9888416Z Traceback (most recent call last): 2024-08-20T23:37:20.9889397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2808, in check_accuracy 2024-08-20T23:37:20.9890343Z optimized_model_iter_fn = optimize_ctx( 2024-08-20T23:37:20.9891238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1400, in export_aot_inductor 2024-08-20T23:37:20.9892255Z optimized = AOTInductorModelCache.load(model, example_inputs, device) 2024-08-20T23:37:20.9893087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1370, in load 2024-08-20T23:37:20.9893796Z gm = torch.export._trace._export_to_torch_ir( 2024-08-20T23:37:20.9894974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/export/_trace.py", line 552, in _export_to_torch_ir 2024-08-20T23:37:20.9895835Z gm_torch_level, _ = torch._dynamo.export( 2024-08-20T23:37:20.9896760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1574, in inner 2024-08-20T23:37:20.9897809Z graph = rewrite_signature( 2024-08-20T23:37:20.9898721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1077, in rewrite_signature 2024-08-20T23:37:20.9899729Z check_user_input_output(flat_results_traced, UserErrorType.INVALID_OUTPUT) 2024-08-20T23:37:20.9900882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1063, in check_user_input_output 2024-08-20T23:37:20.9901734Z raise UserError( 2024-08-20T23:37:20.9902911Z torch._dynamo.exc.UserError: It looks like one of the outputs with type `` is not supported or pytree-flattenable. 2024-08-20T23:37:20.9906261Z Exported graphs outputs can only contain the following supported types: [, , , , , , , , , , , , , , , , , , ]. 2024-08-20T23:37:20.9909266Z If you are using a custom class object, please register a pytree_flatten/unflatten function using `torch.utils._pytree.register_pytree_node` or `torch.export.register_dataclass`. 2024-08-20T23:37:20.9910509Z TorchDynamo optimized model failed to run because of following error 2024-08-20T23:37:20.9935452Z fail_to_run 2024-08-20T23:37:20.9936096Z TIMING: backend_compile:8e-05 entire_frame_compile:0.24502 total_wall_time:0.24502 2024-08-20T23:37:20.9937003Z STATS: call_* op count: 29 | FakeTensorMode.__torch_dispatch__:236 | FakeTensor.__torch_dispatch__:64 2024-08-20T23:37:20.9937877Z Dynamo produced 1 graphs covering 29 ops with 0 graph breaks (0 unique) 2024-08-20T23:37:24.6871737Z 2024-08-20T23:37:25.5830763Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:25.5831657Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:25.5847639Z cuda eval functorch_dp_cifar10 2024-08-20T23:37:44.1751545Z pass 2024-08-20T23:37:44.1787784Z TIMING: backend_compile:0.0001 entire_frame_compile:0.39702 total_wall_time:0.39702 2024-08-20T23:37:44.1788943Z STATS: call_* op count: 69 | FakeTensor.__torch_dispatch__:1056 | FakeTensorMode.__torch_dispatch__:3995 | ProxyTorchDispatchMode.__torch_dispatch__:1397 2024-08-20T23:37:44.1790035Z Dynamo produced 1 graphs covering 69 ops with 0 graph breaks (0 unique) 2024-08-20T23:37:48.2565978Z 2024-08-20T23:37:48.8044743Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:48.8045435Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:48.8048324Z cuda eval functorch_maml_omniglot 2024-08-20T23:38:02.1521738Z pass 2024-08-20T23:38:02.1524996Z TIMING: backend_compile:0.0001 entire_frame_compile:0.16284 total_wall_time:0.16284 2024-08-20T23:38:02.1526328Z STATS: call_* op count: 14 | FakeTensor.__torch_dispatch__:172 | FakeTensorMode.__torch_dispatch__:1419 | ProxyTorchDispatchMode.__torch_dispatch__:282 2024-08-20T23:38:02.1527471Z Dynamo produced 1 graphs covering 14 ops with 0 graph breaks (0 unique) 2024-08-20T23:38:06.0465769Z 2024-08-20T23:38:07.3478565Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:38:07.3479281Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:38:07.3487081Z cuda eval hf_Albert 2024-08-20T23:38:32.3550624Z pass 2024-08-20T23:38:32.3711800Z TIMING: backend_compile:0.00016 entire_frame_compile:1.54822 total_wall_time:1.54822 2024-08-20T23:38:32.3714305Z STATS: call_* op count: 436 | FakeTensorMode.__torch_dispatch__:17352 | FakeTensor.__torch_dispatch__:2978 | ProxyTorchDispatchMode.__torch_dispatch__:6078 2024-08-20T23:38:32.3715437Z Dynamo produced 1 graphs covering 436 ops with 0 graph breaks (0 unique) 2024-08-20T23:38:36.6129329Z 2024-08-20T23:38:40.5647350Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:38:40.5648086Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:38:40.5708571Z cuda eval hf_Bart 2024-08-20T23:39:18.9360230Z pass 2024-08-20T23:39:18.9545343Z TIMING: backend_compile:0.00016 entire_frame_compile:1.98229 total_wall_time:1.98229 2024-08-20T23:39:18.9549115Z STATS: call_* op count: 451 | FakeTensorMode.__torch_dispatch__:21984 | FakeTensor.__torch_dispatch__:3875 | ProxyTorchDispatchMode.__torch_dispatch__:7744 2024-08-20T23:39:18.9550242Z Dynamo produced 1 graphs covering 451 ops with 0 graph breaks (0 unique) 2024-08-20T23:39:23.3548897Z 2024-08-20T23:39:25.9033005Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:25.9033465Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:39:25.9082259Z cuda eval hf_Bert 2024-08-20T23:39:57.2740252Z pass 2024-08-20T23:39:57.2924359Z TIMING: backend_compile:0.00016 entire_frame_compile:3.03313 total_wall_time:3.03313 2024-08-20T23:39:57.2925575Z STATS: call_* op count: 367 | FakeTensorMode.__torch_dispatch__:18390 | FakeTensor.__torch_dispatch__:2959 | ProxyTorchDispatchMode.__torch_dispatch__:6652 2024-08-20T23:39:57.2926690Z Dynamo produced 1 graphs covering 367 ops with 0 graph breaks (0 unique) 2024-08-20T23:40:01.5936664Z 2024-08-20T23:40:06.3372348Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:40:06.3372884Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:40:06.3470831Z cuda eval hf_Bert_large 2024-08-20T23:41:02.3452009Z pass 2024-08-20T23:41:02.3810256Z TIMING: backend_compile:0.00024 entire_frame_compile:5.77309 total_wall_time:5.77309 2024-08-20T23:41:02.3811448Z STATS: call_* op count: 715 | FakeTensorMode.__torch_dispatch__:35586 | FakeTensor.__torch_dispatch__:5755 | ProxyTorchDispatchMode.__torch_dispatch__:13012 2024-08-20T23:41:02.3812667Z Dynamo produced 1 graphs covering 715 ops with 0 graph breaks (0 unique) 2024-08-20T23:41:07.0618667Z 2024-08-20T23:41:10.2997149Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:41:10.2998527Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:41:10.3048223Z cuda eval hf_BigBird 2024-08-20T23:42:05.6662753Z W0820 23:42:05.665000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:06.5977530Z W0820 23:42:06.597000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:07.1304949Z W0820 23:42:07.129000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:07.6604277Z W0820 23:42:07.659000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:08.1891466Z W0820 23:42:08.188000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:08.7213885Z W0820 23:42:08.720000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:09.2660715Z W0820 23:42:09.265000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:09.8081334Z W0820 23:42:09.807000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:10.3514787Z W0820 23:42:10.350000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:10.8866440Z W0820 23:42:10.885000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:11.4227827Z W0820 23:42:11.422000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:42:11.9600734Z W0820 23:42:11.959000 39888 torch/_inductor/utils.py:1391] DeviceCopy in input program 2024-08-20T23:43:23.7758563Z /tmp/tmp0pcs0o3o/cspff2km76arhmvjnpqa2ojqdym7nmgquknjcsniostplznlsejv/c47o2wrpxmf7lhtwx4kkpz3ukp4liwb3nimj7qe3bmx3ebipei5j.cpp: In member function ‘void torch::aot_inductor::AOTInductorModel::run_impl(AtenTensorOpaque**, AtenTensorOpaque**, torch::aot_inductor::DeviceStreamType, AOTIProxyExecutorHandle)’: 2024-08-20T23:43:23.7761880Z /tmp/tmp0pcs0o3o/cspff2km76arhmvjnpqa2ojqdym7nmgquknjcsniostplznlsejv/c47o2wrpxmf7lhtwx4kkpz3ukp4liwb3nimj7qe3bmx3ebipei5j.cpp:4869:10: warning: variable ‘L__self___bert_pooler_weight’ set but not used [-Wunused-but-set-variable] 2024-08-20T23:43:23.7764207Z 4869 | auto L__self___bert_pooler_weight = constants_->at(197); 2024-08-20T23:43:23.7764769Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-08-20T23:43:23.7766388Z /tmp/tmp0pcs0o3o/cspff2km76arhmvjnpqa2ojqdym7nmgquknjcsniostplznlsejv/c47o2wrpxmf7lhtwx4kkpz3ukp4liwb3nimj7qe3bmx3ebipei5j.cpp:4870:10: warning: variable ‘L__self___bert_pooler_bias’ set but not used [-Wunused-but-set-variable] 2024-08-20T23:43:23.7767976Z 4870 | auto L__self___bert_pooler_bias = constants_->at(198); 2024-08-20T23:43:23.7768511Z | ^~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-08-20T23:43:28.5878060Z fail_accuracy 2024-08-20T23:43:28.6168597Z TIMING: backend_compile:0.00069 entire_frame_compile:6.30919 total_wall_time:6.30919 2024-08-20T23:43:28.6169922Z STATS: call_* op count: 3347 | FakeTensorMode.__torch_dispatch__:99344 | FakeTensor.__torch_dispatch__:10152 | ProxyTorchDispatchMode.__torch_dispatch__:30630 2024-08-20T23:43:28.6171078Z Dynamo produced 1 graphs covering 3347 ops with 0 graph breaks (0 unique) 2024-08-20T23:43:35.3292556Z 2024-08-20T23:43:37.4353448Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:43:37.4353925Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:43:37.4378397Z cuda eval hf_DistilBert 2024-08-20T23:44:00.6948362Z pass 2024-08-20T23:44:00.7109398Z TIMING: backend_compile:0.00011 entire_frame_compile:0.90342 total_wall_time:0.90342 2024-08-20T23:44:00.7110853Z STATS: call_* op count: 203 | FakeTensor.__torch_dispatch__:1710 | FakeTensorMode.__torch_dispatch__:10473 | ProxyTorchDispatchMode.__torch_dispatch__:3595 2024-08-20T23:44:00.7111958Z Dynamo produced 1 graphs covering 203 ops with 0 graph breaks (0 unique) 2024-08-20T23:44:04.8414547Z 2024-08-20T23:44:08.1007876Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:44:08.1008590Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:44:08.1045931Z cuda eval hf_GPT2 2024-08-20T23:44:42.4055411Z pass 2024-08-20T23:44:42.4253196Z TIMING: backend_compile:0.00023 entire_frame_compile:2.50336 total_wall_time:2.50336 2024-08-20T23:44:42.4256408Z STATS: call_* op count: 622 | FakeTensorMode.__torch_dispatch__:16933 | FakeTensor.__torch_dispatch__:2394 | ProxyTorchDispatchMode.__torch_dispatch__:5857 2024-08-20T23:44:42.4257547Z Dynamo produced 1 graphs covering 622 ops with 0 graph breaks (0 unique) 2024-08-20T23:44:46.7997748Z 2024-08-20T23:45:00.0886088Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:45:00.0886566Z loading model: 0it [00:13, ?it/s] 2024-08-20T23:45:00.0998857Z cuda eval hf_GPT2_large 2024-08-20T23:45:00.1002581Z pass_due_to_skip 2024-08-20T23:45:00.4405631Z TIMING: 2024-08-20T23:45:00.4406009Z STATS: call_* op count: 0 2024-08-20T23:45:00.4406585Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:45:04.0687021Z 2024-08-20T23:45:06.0719323Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:45:06.0719801Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:45:06.0756362Z cuda eval hf_T5 2024-08-20T23:45:50.0821727Z pass 2024-08-20T23:45:50.1363785Z TIMING: backend_compile:0.00026 entire_frame_compile:3.43089 total_wall_time:3.43089 2024-08-20T23:45:50.1365637Z STATS: call_* op count: 827 | FakeTensorMode.__torch_dispatch__:23837 | FakeTensor.__torch_dispatch__:3262 | ProxyTorchDispatchMode.__torch_dispatch__:8288 2024-08-20T23:45:50.1367630Z Dynamo produced 1 graphs covering 827 ops with 0 graph breaks (0 unique) 2024-08-20T23:45:54.5813478Z 2024-08-20T23:45:58.6774278Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:45:58.6774858Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:45:58.6775403Z cuda eval hf_T5_base 2024-08-20T23:45:58.6788327Z Traceback (most recent call last): 2024-08-20T23:45:58.6789284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:45:58.6790156Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:45:58.6791159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:45:58.6792014Z return mod(*inputs) 2024-08-20T23:45:58.6793387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:45:58.6794358Z return self._call_impl(*args, **kwargs) 2024-08-20T23:45:58.6795324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:45:58.6796410Z return forward_call(*args, **kwargs) 2024-08-20T23:45:58.6797987Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-08-20T23:45:58.6799228Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-08-20T23:45:58.6800407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:45:58.6801297Z return self._call_impl(*args, **kwargs) 2024-08-20T23:45:58.6802233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:45:58.6803070Z return forward_call(*args, **kwargs) 2024-08-20T23:45:58.6804014Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-08-20T23:45:58.6805132Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-08-20T23:45:58.6806277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:45:58.6807209Z return self._call_impl(*args, **kwargs) 2024-08-20T23:45:58.6808127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:45:58.6808944Z return forward_call(*args, **kwargs) 2024-08-20T23:45:58.6810309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1748, in forward 2024-08-20T23:45:58.6811320Z decoder_outputs = self.decoder( 2024-08-20T23:45:58.6812387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:45:58.6813247Z return self._call_impl(*args, **kwargs) 2024-08-20T23:45:58.6814169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:45:58.6814992Z return forward_call(*args, **kwargs) 2024-08-20T23:45:58.6815944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1115, in forward 2024-08-20T23:45:58.6816775Z layer_outputs = layer_module( 2024-08-20T23:45:58.6817714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:45:58.6818585Z return self._call_impl(*args, **kwargs) 2024-08-20T23:45:58.6819511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:45:58.6820322Z return forward_call(*args, **kwargs) 2024-08-20T23:45:58.6821271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 695, in forward 2024-08-20T23:45:58.6822129Z self_attention_outputs = self.layer[0]( 2024-08-20T23:45:58.6823110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:45:58.6823974Z return self._call_impl(*args, **kwargs) 2024-08-20T23:45:58.6824896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:45:58.6825710Z return forward_call(*args, **kwargs) 2024-08-20T23:45:58.6826661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2024-08-20T23:45:58.6827528Z attention_output = self.SelfAttention( 2024-08-20T23:45:58.6828512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:45:58.6829385Z return self._call_impl(*args, **kwargs) 2024-08-20T23:45:58.6830309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:45:58.6831123Z return forward_call(*args, **kwargs) 2024-08-20T23:45:58.6832065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 562, in forward 2024-08-20T23:45:58.6833128Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2024-08-20T23:45:58.6836459Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 58.44 MiB is free. Process 96516 has 21.91 GiB memory in use. Of the allocated memory 20.94 GiB is allocated by PyTorch, and 666.49 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T23:45:58.6839174Z 2024-08-20T23:45:58.6839468Z The above exception was the direct cause of the following exception: 2024-08-20T23:45:58.6839916Z 2024-08-20T23:45:58.6840060Z Traceback (most recent call last): 2024-08-20T23:45:58.6840706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:45:58.6841353Z ) = runner.load_model( 2024-08-20T23:45:58.6842001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:45:58.6842751Z self.validate_model(model, example_inputs) 2024-08-20T23:45:58.6843496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:45:58.6844356Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:45:58.6844894Z RuntimeError: Eager run failed 2024-08-20T23:45:58.6845153Z 2024-08-20T23:45:58.6845269Z eager_fail_to_run 2024-08-20T23:46:02.3472523Z 2024-08-20T23:46:12.3015914Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:12.3016487Z loading model: 0it [00:09, ?it/s] 2024-08-20T23:46:12.3409223Z cuda eval hf_T5_large 2024-08-20T23:46:12.3412841Z pass_due_to_skip 2024-08-20T23:46:12.6379788Z TIMING: 2024-08-20T23:46:12.6380107Z STATS: call_* op count: 0 2024-08-20T23:46:12.6380658Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:46:16.2450417Z 2024-08-20T23:46:17.3636199Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:17.3636657Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:46:17.3655099Z cuda eval hf_Whisper 2024-08-20T23:46:36.7503368Z pass 2024-08-20T23:46:36.7571227Z TIMING: backend_compile:0.00011 entire_frame_compile:0.56991 total_wall_time:0.56991 2024-08-20T23:46:36.7573663Z STATS: call_* op count: 115 | FakeTensor.__torch_dispatch__:1108 | FakeTensorMode.__torch_dispatch__:6056 | ProxyTorchDispatchMode.__torch_dispatch__:2046 2024-08-20T23:46:36.7575863Z Dynamo produced 1 graphs covering 115 ops with 0 graph breaks (0 unique) 2024-08-20T23:46:40.8424786Z 2024-08-20T23:46:45.5986232Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:45.5986694Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:46:45.6081950Z cuda eval hf_distil_whisper 2024-08-20T23:48:01.9529224Z pass 2024-08-20T23:48:01.9800567Z TIMING: backend_compile:0.0002 entire_frame_compile:2.58502 total_wall_time:2.58502 2024-08-20T23:48:01.9801751Z STATS: call_* op count: 635 | FakeTensor.__torch_dispatch__:5948 | FakeTensorMode.__torch_dispatch__:31312 | ProxyTorchDispatchMode.__torch_dispatch__:11226 2024-08-20T23:48:01.9802863Z Dynamo produced 1 graphs covering 635 ops with 0 graph breaks (0 unique) 2024-08-20T23:48:06.7172187Z 2024-08-20T23:48:07.1485328Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:07.1485956Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:07.1486455Z cuda eval lennard_jones 2024-08-20T23:48:19.6771231Z pass 2024-08-20T23:48:19.6775557Z TIMING: backend_compile:8e-05 entire_frame_compile:0.11554 total_wall_time:0.11554 2024-08-20T23:48:19.6777072Z STATS: call_* op count: 9 | FakeTensor.__torch_dispatch__:82 | FakeTensorMode.__torch_dispatch__:933 | ProxyTorchDispatchMode.__torch_dispatch__:238 2024-08-20T23:48:19.6778297Z Dynamo produced 1 graphs covering 9 ops with 0 graph breaks (0 unique) 2024-08-20T23:48:20.6694212Z accuracy pass_rate=51.16% 2024-08-20T23:48:20.6695026Z calls_captured gmean=0.00x mean=212.814x 2024-08-20T23:48:20.6698996Z unique_graphs gmean=0.00x mean=0.581x 2024-08-20T23:48:20.6703323Z graph_breaks gmean=0.00x mean=0.000x 2024-08-20T23:48:20.6706279Z unique_graph_breaks gmean=0.00x mean=0.000x 2024-08-20T23:48:20.6709629Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:48:20.6712962Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:48:20.6716242Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T23:48:22.9024893Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_inductor_torchbench_inference.csv 2024-08-20T23:48:23.2462159Z torchrec_dlrm XFAIL 2024-08-20T23:48:23.2465844Z BERT_pytorch XFAIL 2024-08-20T23:48:23.2469985Z Background_Matting XFAIL 2024-08-20T23:48:23.2474256Z LearningToPaint PASS 2024-08-20T23:48:23.2478668Z Super_SloMo PASS 2024-08-20T23:48:23.2483219Z alexnet PASS 2024-08-20T23:48:23.2487549Z basic_gnn_edgecnn PASS 2024-08-20T23:48:23.2491770Z basic_gnn_gcn XFAIL 2024-08-20T23:48:23.2496843Z basic_gnn_gin PASS 2024-08-20T23:48:23.2501676Z basic_gnn_sage PASS 2024-08-20T23:48:23.2505858Z dcgan PASS 2024-08-20T23:48:23.2510399Z demucs PASS 2024-08-20T23:48:23.2514741Z densenet121 PASS 2024-08-20T23:48:23.2519303Z detectron2_fasterrcnn_r_101_c4 XFAIL 2024-08-20T23:48:23.2523625Z detectron2_fasterrcnn_r_101_dc5 XFAIL 2024-08-20T23:48:23.2528105Z detectron2_fasterrcnn_r_101_fpn XFAIL 2024-08-20T23:48:23.2532523Z detectron2_fasterrcnn_r_50_c4 XFAIL 2024-08-20T23:48:23.2537238Z detectron2_fasterrcnn_r_50_dc5 XFAIL 2024-08-20T23:48:23.2541357Z detectron2_fasterrcnn_r_50_fpn XFAIL 2024-08-20T23:48:23.2545839Z detectron2_maskrcnn_r_101_c4 XFAIL 2024-08-20T23:48:23.2550186Z detectron2_maskrcnn_r_101_fpn XFAIL 2024-08-20T23:48:23.2554761Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-08-20T23:48:23.2558906Z detectron2_maskrcnn_r_50_fpn XFAIL 2024-08-20T23:48:23.2563348Z dlrm PASS 2024-08-20T23:48:23.2567705Z doctr_det_predictor XFAIL 2024-08-20T23:48:23.2572185Z doctr_reco_predictor XFAIL 2024-08-20T23:48:23.2576739Z drq XFAIL 2024-08-20T23:48:23.2581090Z functorch_dp_cifar10 PASS 2024-08-20T23:48:23.2585431Z functorch_maml_omniglot PASS 2024-08-20T23:48:23.2590175Z hf_Albert PASS 2024-08-20T23:48:23.2594454Z hf_Bart PASS 2024-08-20T23:48:23.2599336Z hf_Bert PASS 2024-08-20T23:48:23.2603956Z hf_Bert_large PASS 2024-08-20T23:48:23.2608287Z hf_BigBird XFAIL 2024-08-20T23:48:23.2612787Z hf_DistilBert PASS 2024-08-20T23:48:23.2617419Z hf_GPT2 PASS 2024-08-20T23:48:23.2621933Z hf_GPT2_large XFAIL 2024-08-20T23:48:23.2626137Z hf_T5 PASS 2024-08-20T23:48:23.2630861Z hf_T5_base XFAIL 2024-08-20T23:48:23.2635135Z hf_T5_large XFAIL 2024-08-20T23:48:23.2639573Z hf_Whisper PASS 2024-08-20T23:48:23.2643723Z hf_distil_whisper PASS 2024-08-20T23:48:23.2648083Z lennard_jones PASS 2024-08-20T23:48:23.3178603Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_inductor_torchbench_inference.csv 2024-08-20T23:48:23.6557929Z torchrec_dlrm PASS 2024-08-20T23:48:23.6561596Z BERT_pytorch PASS 2024-08-20T23:48:23.6566842Z Background_Matting PASS 2024-08-20T23:48:23.6571581Z LearningToPaint PASS 2024-08-20T23:48:23.6576908Z Super_SloMo PASS 2024-08-20T23:48:23.6581472Z alexnet PASS 2024-08-20T23:48:23.6586317Z basic_gnn_edgecnn PASS 2024-08-20T23:48:23.6591165Z basic_gnn_gcn PASS 2024-08-20T23:48:23.6596235Z basic_gnn_gin PASS 2024-08-20T23:48:23.6601529Z basic_gnn_sage PASS 2024-08-20T23:48:23.6606339Z dcgan PASS 2024-08-20T23:48:23.6611193Z demucs PASS 2024-08-20T23:48:23.6616460Z densenet121 PASS 2024-08-20T23:48:23.6621246Z detectron2_fasterrcnn_r_101_c4 PASS 2024-08-20T23:48:23.6626140Z detectron2_fasterrcnn_r_101_dc5 PASS 2024-08-20T23:48:23.6631080Z detectron2_fasterrcnn_r_101_fpn PASS 2024-08-20T23:48:23.6636061Z detectron2_fasterrcnn_r_50_c4 PASS 2024-08-20T23:48:23.6640921Z detectron2_fasterrcnn_r_50_dc5 PASS 2024-08-20T23:48:23.6645679Z detectron2_fasterrcnn_r_50_fpn PASS 2024-08-20T23:48:23.6650543Z detectron2_maskrcnn_r_101_c4 PASS 2024-08-20T23:48:23.6655695Z detectron2_maskrcnn_r_101_fpn PASS 2024-08-20T23:48:23.6660613Z detectron2_maskrcnn_r_50_c4 PASS 2024-08-20T23:48:23.6665318Z detectron2_maskrcnn_r_50_fpn PASS 2024-08-20T23:48:23.6670192Z dlrm PASS 2024-08-20T23:48:23.6675091Z doctr_det_predictor PASS 2024-08-20T23:48:23.6679926Z doctr_reco_predictor PASS 2024-08-20T23:48:23.6684659Z drq PASS 2024-08-20T23:48:23.6689596Z functorch_dp_cifar10 PASS 2024-08-20T23:48:23.6694784Z functorch_maml_omniglot PASS 2024-08-20T23:48:23.6699765Z hf_Albert PASS 2024-08-20T23:48:23.6704739Z hf_Bart PASS 2024-08-20T23:48:23.6709589Z hf_Bert PASS 2024-08-20T23:48:23.6714584Z hf_Bert_large PASS 2024-08-20T23:48:23.6719326Z hf_BigBird PASS 2024-08-20T23:48:23.6724358Z hf_DistilBert PASS 2024-08-20T23:48:23.6729039Z hf_GPT2 PASS 2024-08-20T23:48:23.6734250Z hf_GPT2_large PASS 2024-08-20T23:48:23.6739047Z hf_T5 PASS 2024-08-20T23:48:23.6743886Z hf_T5_base PASS 2024-08-20T23:48:23.6748602Z hf_T5_large PASS 2024-08-20T23:48:23.6753503Z hf_Whisper PASS 2024-08-20T23:48:23.6758273Z hf_distil_whisper PASS 2024-08-20T23:48:23.6763246Z lennard_jones PASS 2024-08-20T23:48:23.7298864Z + cleanup_workspace 2024-08-20T23:48:23.7299892Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:48:23.7300982Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:48:23.7301910Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:48:23.7302557Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:48:23.7303365Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:48:23.7304320Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:48:23.7305266Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:48:24.4966994Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:48:24.4967517Z cat test/**/*_toprint.log || true 2024-08-20T23:48:24.4981158Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:24.4981656Z env: 2024-08-20T23:48:24.4981936Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:24.4982396Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:24.4983126Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:24.4983761Z ##[endgroup] 2024-08-20T23:48:24.5082338Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:48:24.5115021Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:48:24.5115482Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:48:24.5124011Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:24.5124510Z env: 2024-08-20T23:48:24.5124789Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:24.5125244Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:24.5125965Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:24.5126612Z MONITOR_SCRIPT_PID: 52484 2024-08-20T23:48:24.5126966Z ##[endgroup] 2024-08-20T23:48:24.5330725Z Prepare all required actions 2024-08-20T23:48:24.5331168Z Getting action download info 2024-08-20T23:48:24.7267140Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:48:24.8870718Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:48:24.8871181Z with: 2024-08-20T23:48:24.8871972Z file-suffix: test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597 2024-08-20T23:48:24.8872691Z s3-bucket: gha-artifacts 2024-08-20T23:48:24.8873032Z env: 2024-08-20T23:48:24.8873302Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:24.8873751Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:24.8874481Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:24.8875115Z ##[endgroup] 2024-08-20T23:48:24.8907379Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:48:24.8908013Z # Remove any previous test jsons if they exist 2024-08-20T23:48:24.8908524Z rm -f test-jsons-*.zip 2024-08-20T23:48:24.8909073Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:48:24.8918070Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:24.8918563Z env: 2024-08-20T23:48:24.8918841Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:24.8919302Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:24.8920026Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:24.8920960Z FILE_SUFFIX: test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597 2024-08-20T23:48:24.8921650Z ##[endgroup] 2024-08-20T23:48:24.9708501Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:48:24.9748949Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:48:24.9755426Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:48:24.9771390Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:48:24.9785231Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:48:25.0449443Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:48:25.0450145Z # Remove any previous test reports if they exist 2024-08-20T23:48:25.0450663Z rm -f test-reports-*.zip 2024-08-20T23:48:25.0451259Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:48:25.0460643Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:25.0461137Z env: 2024-08-20T23:48:25.0461420Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:25.0461873Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:25.0462591Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:25.0463547Z FILE_SUFFIX: test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597 2024-08-20T23:48:25.0464241Z ##[endgroup] 2024-08-20T23:48:25.0722930Z adding: test/test-reports/inference_torchbench.csv (deflated 75%) 2024-08-20T23:48:25.0753735Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:48:25.0754352Z # Remove any previous usage logs if they exist 2024-08-20T23:48:25.0754841Z rm -f logs-*.zip 2024-08-20T23:48:25.0755506Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:48:25.0756281Z # so check to see if the file exists first 2024-08-20T23:48:25.0756772Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:48:25.0757292Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:48:25.0757773Z fi 2024-08-20T23:48:25.0758130Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:48:25.0758689Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:48:25.0759175Z fi 2024-08-20T23:48:25.0767327Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:25.0767820Z env: 2024-08-20T23:48:25.0768088Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:25.0768536Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:25.0769255Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:25.0770343Z FILE_SUFFIX: test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597 2024-08-20T23:48:25.0771029Z ##[endgroup] 2024-08-20T23:48:25.0864841Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:48:25.0951436Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:48:25.0952140Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:48:25.0952681Z rm -f debug-*.zip 2024-08-20T23:48:25.0953062Z if [ -d 'test/debug' ]; then 2024-08-20T23:48:25.0953557Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:48:25.0954042Z fi 2024-08-20T23:48:25.0961865Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:25.0962356Z env: 2024-08-20T23:48:25.0962634Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:25.0963084Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:25.0963811Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:25.0964758Z FILE_SUFFIX: test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597 2024-08-20T23:48:25.0965453Z ##[endgroup] 2024-08-20T23:48:25.1058088Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:48:25.1058523Z with: 2024-08-20T23:48:25.1058804Z s3-bucket: gha-artifacts 2024-08-20T23:48:25.1059236Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:48:25.1059709Z retention-days: 14 2024-08-20T23:48:25.1060036Z if-no-files-found: warn 2024-08-20T23:48:25.1060393Z path: test-jsons-*.zip 2024-08-20T23:48:25.1060742Z name: artifact 2024-08-20T23:48:25.1061039Z region: us-east-1 2024-08-20T23:48:25.1061342Z env: 2024-08-20T23:48:25.1061620Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:25.1062068Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:25.1062794Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:25.1063428Z ##[endgroup] 2024-08-20T23:48:25.7007199Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:48:25.7007917Z With the provided path, there will be 1 file uploaded 2024-08-20T23:48:25.7008721Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:48:25.8673647Z Starting upload of test-jsons-test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597.zip 2024-08-20T23:48:25.9792485Z Finished upload of test-jsons-test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597.zip 2024-08-20T23:48:26.0080536Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:48:26.0080989Z with: 2024-08-20T23:48:26.0081276Z s3-bucket: gha-artifacts 2024-08-20T23:48:26.0081711Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:48:26.0082183Z retention-days: 14 2024-08-20T23:48:26.0082513Z if-no-files-found: error 2024-08-20T23:48:26.0082877Z path: test-reports-*.zip 2024-08-20T23:48:26.0083226Z name: artifact 2024-08-20T23:48:26.0083536Z region: us-east-1 2024-08-20T23:48:26.0083840Z env: 2024-08-20T23:48:26.0084115Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:26.0084563Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:26.0085285Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:26.0085917Z ##[endgroup] 2024-08-20T23:48:26.3674416Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:48:26.3675953Z With the provided path, there will be 1 file uploaded 2024-08-20T23:48:26.3677413Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:48:26.3733655Z Starting upload of test-reports-test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597.zip 2024-08-20T23:48:26.4790230Z Finished upload of test-reports-test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597.zip 2024-08-20T23:48:26.5095526Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:48:26.5095962Z with: 2024-08-20T23:48:26.5096434Z s3-bucket: gha-artifacts 2024-08-20T23:48:26.5096866Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:48:26.5097333Z retention-days: 14 2024-08-20T23:48:26.5098075Z if-no-files-found: ignore 2024-08-20T23:48:26.5098434Z path: logs-*.zip 2024-08-20T23:48:26.5098740Z name: artifact 2024-08-20T23:48:26.5099035Z region: us-east-1 2024-08-20T23:48:26.5099332Z env: 2024-08-20T23:48:26.5099602Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:26.5100048Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:26.5100778Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:26.5101403Z ##[endgroup] 2024-08-20T23:48:26.8679789Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:48:26.8680497Z With the provided path, there will be 1 file uploaded 2024-08-20T23:48:26.8681102Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:48:26.8737023Z Starting upload of logs-test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597.zip 2024-08-20T23:48:26.9663404Z Finished upload of logs-test-aot_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026530597.zip 2024-08-20T23:48:26.9950548Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:48:26.9950989Z with: 2024-08-20T23:48:26.9951277Z s3-bucket: gha-artifacts 2024-08-20T23:48:26.9951716Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:48:26.9952193Z retention-days: 14 2024-08-20T23:48:26.9952527Z if-no-files-found: ignore 2024-08-20T23:48:26.9952894Z path: debug-*.zip 2024-08-20T23:48:26.9953224Z name: artifact 2024-08-20T23:48:26.9953529Z region: us-east-1 2024-08-20T23:48:26.9953833Z env: 2024-08-20T23:48:26.9954117Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:26.9954576Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:26.9955298Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:26.9955944Z ##[endgroup] 2024-08-20T23:48:27.3428594Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:48:27.3713490Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:48:27.3713986Z # shellcheck disable=SC2156 2024-08-20T23:48:27.3714771Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:48:27.3723896Z shell: /usr/bin/bash -e {0} 2024-08-20T23:48:27.3724249Z env: 2024-08-20T23:48:27.3724532Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:27.3724984Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:27.3725711Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:27.3726343Z ##[endgroup] 2024-08-20T23:48:27.6394569Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:48:27.6395131Z with: 2024-08-20T23:48:27.6395393Z env: 2024-08-20T23:48:27.6395684Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:27.6396136Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:27.6396870Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:27.6397710Z ##[endgroup] 2024-08-20T23:48:27.6417108Z ##[group]Run set -eou pipefail 2024-08-20T23:48:27.6417512Z set -eou pipefail 2024-08-20T23:48:27.6417858Z  2024-08-20T23:48:27.6418360Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:48:27.6418984Z for _ in $(seq 1440); do 2024-08-20T23:48:27.6419450Z  # Break if no ssh session exists anymore 2024-08-20T23:48:27.6419946Z  if [ "$(who)" = "" ]; then 2024-08-20T23:48:27.6420348Z  break 2024-08-20T23:48:27.6420682Z  fi 2024-08-20T23:48:27.6420973Z  echo "." 2024-08-20T23:48:27.6421297Z  sleep 5 2024-08-20T23:48:27.6421612Z done 2024-08-20T23:48:27.6430527Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:27.6431021Z env: 2024-08-20T23:48:27.6431299Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:27.6431745Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:27.6432468Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:27.6433093Z ##[endgroup] 2024-08-20T23:48:27.6463080Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:48:27.6608685Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:48:27.6609456Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:48:27.6610095Z # shellcheck disable=SC2046 2024-08-20T23:48:27.6610579Z docker stop $(docker ps -q) || true 2024-08-20T23:48:27.6611059Z # Prune all of the docker images 2024-08-20T23:48:27.6611509Z docker system prune -af 2024-08-20T23:48:27.6620502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:27.6620999Z env: 2024-08-20T23:48:27.6621281Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:48:27.6621742Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:48:27.6622463Z DOCKER_CONTAINER_ID: 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:27.6623095Z ##[endgroup] 2024-08-20T23:48:28.3853227Z 7f35680bf802 2024-08-20T23:48:32.4002512Z Deleted Containers: 2024-08-20T23:48:32.4003178Z 7f35680bf8024c5e581a686691a4ca7ff1de44f28fe86d66ae4a3a5243c33586 2024-08-20T23:48:32.4003716Z 2024-08-20T23:48:41.2640428Z Deleted Images: 2024-08-20T23:48:41.2643939Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:48:41.2646490Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:48:41.2648013Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:48:41.2648855Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:48:41.2649708Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:48:41.2650521Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:48:41.2651321Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:48:41.2652285Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:48:41.2653215Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:48:41.2654499Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:48:41.2655332Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:48:41.2656178Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:48:41.2657053Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:48:41.2657859Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:48:41.2658671Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:48:41.2659478Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:48:41.2660286Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:48:41.2661206Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:48:41.2662038Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:48:41.2662845Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:48:41.2663879Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:48:41.2664686Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:48:41.2665502Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:48:41.2666323Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:48:41.2667134Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:48:41.2667939Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:48:41.2668749Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:48:41.2669573Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:48:41.2670392Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:48:41.2671217Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:48:41.2672126Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:48:41.2673111Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:48:41.2674237Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:48:41.2675297Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:48:41.2676202Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:48:41.2677023Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:48:41.2677840Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:48:41.2678659Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:48:41.2679474Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:48:41.2680303Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:48:41.2681124Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:48:41.2681953Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:48:41.2682784Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:48:41.2683630Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:48:41.2684450Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:48:41.2685253Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:48:41.2686238Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:48:41.2687084Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:48:41.2687905Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:48:41.2688694Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:48:41.2689539Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:48:41.2690367Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:48:41.2691186Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:48:41.2692072Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:48:41.2692874Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:48:41.2693698Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:48:41.2694530Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:48:41.2695337Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:48:41.2696247Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:48:41.2697049Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:48:41.2698107Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:48:41.2698908Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:48:41.2699934Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:48:41.2701031Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:48:41.2702118Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:48:41.2703181Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:48:41.2704070Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:48:41.2704907Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:48:41.2705729Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:48:41.2706555Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:48:41.2707375Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:48:41.2708195Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:48:41.2708997Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:48:41.2709804Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:48:41.2710623Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:48:41.2711433Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:48:41.2712265Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:48:41.2713092Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:48:41.2713956Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:48:41.2714793Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:48:41.2715603Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:48:41.2716089Z 2024-08-20T23:48:41.2716250Z Total reclaimed space: 49.69GB 2024-08-20T23:48:41.2782985Z Post job cleanup. 2024-08-20T23:48:41.2833623Z Post job cleanup. 2024-08-20T23:48:41.3751474Z [command]/usr/bin/git version 2024-08-20T23:48:41.3841696Z git version 2.40.1 2024-08-20T23:48:41.3880860Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a1f24111-d87a-493a-8b53-31019121a0ed' before making global git config changes 2024-08-20T23:48:41.3882389Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:48:41.3886618Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:48:41.3935979Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:48:41.3980428Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:48:41.4361933Z Entering 'android/libs/fbjni' 2024-08-20T23:48:41.4431737Z Entering 'third_party/FP16' 2024-08-20T23:48:41.4503302Z Entering 'third_party/FXdiv' 2024-08-20T23:48:41.4574084Z Entering 'third_party/NNPACK' 2024-08-20T23:48:41.4644395Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:48:41.4720890Z Entering 'third_party/XNNPACK' 2024-08-20T23:48:41.4806995Z Entering 'third_party/benchmark' 2024-08-20T23:48:41.4877073Z Entering 'third_party/cpp-httplib' 2024-08-20T23:48:41.4949231Z Entering 'third_party/cpuinfo' 2024-08-20T23:48:41.5019587Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:48:41.5089888Z Entering 'third_party/cutlass' 2024-08-20T23:48:41.5170037Z Entering 'third_party/eigen' 2024-08-20T23:48:41.5242446Z Entering 'third_party/fbgemm' 2024-08-20T23:48:41.5311600Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:48:41.5381273Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:48:41.5447822Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:48:41.5520598Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:48:41.5586489Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:48:41.5656878Z Entering 'third_party/flatbuffers' 2024-08-20T23:48:41.5731721Z Entering 'third_party/fmt' 2024-08-20T23:48:41.5806095Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:48:41.5874671Z Entering 'third_party/gloo' 2024-08-20T23:48:41.5943580Z Entering 'third_party/googletest' 2024-08-20T23:48:41.6015146Z Entering 'third_party/ideep' 2024-08-20T23:48:41.6082078Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:48:41.6159855Z Entering 'third_party/ittapi' 2024-08-20T23:48:41.6231198Z Entering 'third_party/kineto' 2024-08-20T23:48:41.6299947Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:48:41.6366140Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:48:41.6437454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:48:41.6508131Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:48:41.6576723Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:48:41.6642157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:48:41.6714877Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:48:41.6782059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:48:41.6850117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:48:41.6920279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:48:41.6990831Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:48:41.7059245Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:48:41.7129508Z Entering 'third_party/mimalloc' 2024-08-20T23:48:41.7199968Z Entering 'third_party/nccl/nccl' 2024-08-20T23:48:41.7269237Z Entering 'third_party/nlohmann' 2024-08-20T23:48:41.7337394Z Entering 'third_party/onnx' 2024-08-20T23:48:41.7418629Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:48:41.7486863Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:48:41.7560875Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:48:41.7628689Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:48:41.7696238Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:48:41.7763374Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:48:41.7830689Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:48:41.7900188Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:48:41.7967646Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:48:41.8035089Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:48:41.8100629Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:48:41.8168594Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:48:41.8238397Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:48:41.8327457Z Entering 'third_party/pocketfft' 2024-08-20T23:48:41.8393836Z Entering 'third_party/protobuf' 2024-08-20T23:48:41.8464816Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:48:41.8532292Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:48:41.8603979Z Entering 'third_party/psimd' 2024-08-20T23:48:41.8676141Z Entering 'third_party/pthreadpool' 2024-08-20T23:48:41.8747079Z Entering 'third_party/pybind11' 2024-08-20T23:48:41.8817192Z Entering 'third_party/python-peachpy' 2024-08-20T23:48:41.8886886Z Entering 'third_party/sleef' 2024-08-20T23:48:41.8956916Z Entering 'third_party/tensorpipe' 2024-08-20T23:48:41.9024238Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:48:41.9095495Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:48:41.9168697Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:48:41.9234935Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:48:41.9300048Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:48:41.9394892Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:48:41.9430317Z http.https://github.com/.extraheader 2024-08-20T23:48:41.9440239Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:48:41.9486131Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:48:41.9856980Z Entering 'android/libs/fbjni' 2024-08-20T23:48:41.9900416Z http.https://github.com/.extraheader 2024-08-20T23:48:41.9945800Z Entering 'third_party/FP16' 2024-08-20T23:48:41.9990655Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0035365Z Entering 'third_party/FXdiv' 2024-08-20T23:48:42.0080215Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0124885Z Entering 'third_party/NNPACK' 2024-08-20T23:48:42.0169681Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0214771Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:48:42.0259785Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0304998Z Entering 'third_party/XNNPACK' 2024-08-20T23:48:42.0349886Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0410557Z Entering 'third_party/benchmark' 2024-08-20T23:48:42.0455571Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0499344Z Entering 'third_party/cpp-httplib' 2024-08-20T23:48:42.0544630Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0589414Z Entering 'third_party/cpuinfo' 2024-08-20T23:48:42.0633428Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0681822Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:48:42.0726016Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0770385Z Entering 'third_party/cutlass' 2024-08-20T23:48:42.0817058Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0869117Z Entering 'third_party/eigen' 2024-08-20T23:48:42.0912805Z http.https://github.com/.extraheader 2024-08-20T23:48:42.0959251Z Entering 'third_party/fbgemm' 2024-08-20T23:48:42.1004267Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1047747Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:48:42.1090471Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1135442Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:48:42.1182141Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1227124Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:48:42.1269396Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1318574Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:48:42.1361162Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1404485Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:48:42.1451827Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1498228Z Entering 'third_party/flatbuffers' 2024-08-20T23:48:42.1543836Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1590465Z Entering 'third_party/fmt' 2024-08-20T23:48:42.1635629Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1679730Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:48:42.1725654Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1770005Z Entering 'third_party/gloo' 2024-08-20T23:48:42.1815888Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1860532Z Entering 'third_party/googletest' 2024-08-20T23:48:42.1905920Z http.https://github.com/.extraheader 2024-08-20T23:48:42.1950205Z Entering 'third_party/ideep' 2024-08-20T23:48:42.1995587Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2041117Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:48:42.2083541Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2138791Z Entering 'third_party/ittapi' 2024-08-20T23:48:42.2182340Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2226335Z Entering 'third_party/kineto' 2024-08-20T23:48:42.2270042Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2312745Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:48:42.2357299Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2401334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:48:42.2445711Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2490882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:48:42.2533712Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2577569Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:48:42.2620539Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2664308Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:48:42.2707341Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2750615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:48:42.2795056Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2844142Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:48:42.2886567Z http.https://github.com/.extraheader 2024-08-20T23:48:42.2931795Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:48:42.2974777Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3022757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:48:42.3065398Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3111041Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:48:42.3154685Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3202467Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:48:42.3246074Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3289592Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:48:42.3334864Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3381086Z Entering 'third_party/mimalloc' 2024-08-20T23:48:42.3425991Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3470093Z Entering 'third_party/nccl/nccl' 2024-08-20T23:48:42.3514776Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3559124Z Entering 'third_party/nlohmann' 2024-08-20T23:48:42.3603050Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3647926Z Entering 'third_party/onnx' 2024-08-20T23:48:42.3691707Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3750454Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:48:42.3792933Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3840706Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:48:42.3883491Z http.https://github.com/.extraheader 2024-08-20T23:48:42.3932424Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:48:42.3977234Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4022628Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:48:42.4064873Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4109901Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:48:42.4151954Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4195689Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:48:42.4238858Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4281567Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:48:42.4324724Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4369645Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:48:42.4411724Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4454424Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:48:42.4496441Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4540551Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:48:42.4583021Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4625255Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:48:42.4672021Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4718185Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:48:42.4760988Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4809144Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:48:42.4851878Z http.https://github.com/.extraheader 2024-08-20T23:48:42.4918291Z Entering 'third_party/pocketfft' 2024-08-20T23:48:42.4963388Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5007666Z Entering 'third_party/protobuf' 2024-08-20T23:48:42.5052024Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5098113Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:48:42.5140848Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5182857Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:48:42.5226654Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5274271Z Entering 'third_party/psimd' 2024-08-20T23:48:42.5319702Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5364099Z Entering 'third_party/pthreadpool' 2024-08-20T23:48:42.5409775Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5454011Z Entering 'third_party/pybind11' 2024-08-20T23:48:42.5498091Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5542546Z Entering 'third_party/python-peachpy' 2024-08-20T23:48:42.5587307Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5631558Z Entering 'third_party/sleef' 2024-08-20T23:48:42.5675436Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5722259Z Entering 'third_party/tensorpipe' 2024-08-20T23:48:42.5766516Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5810268Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:48:42.5852237Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5896315Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:48:42.5946078Z http.https://github.com/.extraheader 2024-08-20T23:48:42.5989390Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:48:42.6031712Z http.https://github.com/.extraheader 2024-08-20T23:48:42.6074910Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:48:42.6118173Z http.https://github.com/.extraheader 2024-08-20T23:48:42.6159054Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:48:42.6203847Z http.https://github.com/.extraheader 2024-08-20T23:48:42.6364717Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:48:42.6384870Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:48:42.6392876Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:48:42.6393382Z ##[endgroup] 2024-08-20T23:48:49.6372386Z Cleaning up orphan processes