2024-08-20T22:09:47.6893445Z Current runner version: '2.319.1' 2024-08-20T22:09:47.6899620Z Runner name: 'i-0284de2732bb465d7' 2024-08-20T22:09:47.6900767Z Runner group name: 'Default' 2024-08-20T22:09:47.6901626Z Machine name: 'ip-10-0-11-223' 2024-08-20T22:09:47.6918046Z Testing runner upgrade compatibility 2024-08-20T22:09:47.8443232Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:09:47.8445352Z Actions: read 2024-08-20T22:09:47.8445813Z Attestations: read 2024-08-20T22:09:47.8446329Z Checks: read 2024-08-20T22:09:47.8446877Z Contents: read 2024-08-20T22:09:47.8447353Z Deployments: read 2024-08-20T22:09:47.8447827Z Discussions: read 2024-08-20T22:09:47.8448353Z Issues: read 2024-08-20T22:09:47.8448800Z Metadata: read 2024-08-20T22:09:47.8449246Z Packages: read 2024-08-20T22:09:47.8449761Z Pages: read 2024-08-20T22:09:47.8450209Z PullRequests: read 2024-08-20T22:09:47.8450784Z RepositoryProjects: read 2024-08-20T22:09:47.8451297Z SecurityEvents: read 2024-08-20T22:09:47.8451812Z Statuses: read 2024-08-20T22:09:47.8452493Z ##[endgroup] 2024-08-20T22:09:47.8455643Z Secret source: Actions 2024-08-20T22:09:47.8456357Z Prepare workflow directory 2024-08-20T22:09:47.9366635Z Prepare all required actions 2024-08-20T22:09:47.9530624Z Getting action download info 2024-08-20T22:09:48.0976936Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:09:48.5941714Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:09:52.6479911Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:09:52.8602109Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:09:53.1654397Z Getting action download info 2024-08-20T22:09:53.2505496Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:09:53.4429067Z Getting action download info 2024-08-20T22:09:53.5393967Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:09:53.6936341Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:09:53.6938473Z ##[group] Inputs 2024-08-20T22:09:53.6938984Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:09:53.6948536Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:09:53.6959094Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:09:53.6960294Z sync-tag: 2024-08-20T22:09:53.6961231Z timeout-minutes: 240 2024-08-20T22:09:53.6961582Z use-gha: 2024-08-20T22:09:53.6961885Z dashboard-tag: 2024-08-20T22:09:53.6962219Z s3-bucket: gha-artifacts 2024-08-20T22:09:53.6962583Z aws-role-to-assume: 2024-08-20T22:09:53.6962927Z ##[endgroup] 2024-08-20T22:09:53.6963887Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:09:53.7690814Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:09:53.7832442Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:09:53.7844313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:09:53.7844854Z ##[endgroup] 2024-08-20T22:09:55.0008227Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:09:55.0008835Z Instance Type: g5.4xlarge 2024-08-20T22:09:55.0009534Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:09:55.0010075Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:00.7864630Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:00.7865188Z with: 2024-08-20T22:10:00.7865961Z github-secret: *** 2024-08-20T22:10:00.7866962Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:00.7867979Z activate-with-label: false 2024-08-20T22:10:00.7868356Z label: with-ssh 2024-08-20T22:10:00.7868691Z remove-existing-keys: true 2024-08-20T22:10:00.7869071Z fail-silently: true 2024-08-20T22:10:00.7869389Z env: 2024-08-20T22:10:00.7869687Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.7870047Z ##[endgroup] 2024-08-20T22:10:00.8772034Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:00.8774926Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:01.1927485Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:01.2664719Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:01.2665356Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:01.3770500Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:01.3784572Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:01.3808777Z Login using: ssh ec2-user@ec2-54-89-3-124.compute-1.amazonaws.com 2024-08-20T22:10:01.3809633Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:01.3810600Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:01.3940727Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:01.3941295Z with: 2024-08-20T22:10:01.3941576Z submodules: recursive 2024-08-20T22:10:01.3941918Z fetch-depth: 0 2024-08-20T22:10:01.3942210Z env: 2024-08-20T22:10:01.3942478Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:01.3942822Z ##[endgroup] 2024-08-20T22:10:01.4132438Z ##[group]Run retry () { 2024-08-20T22:10:01.4132830Z retry () { 2024-08-20T22:10:01.4133543Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:01.4134103Z } 2024-08-20T22:10:01.4134415Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:01.4134843Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:01.4135326Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:01.4135792Z else 2024-08-20T22:10:01.4136140Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:01.4136574Z fi 2024-08-20T22:10:01.4136923Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:01.4150339Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:01.4150856Z env: 2024-08-20T22:10:01.4151144Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:01.4151489Z NO_SUDO: 2024-08-20T22:10:01.4151768Z ##[endgroup] 2024-08-20T22:10:01.4185249Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:01.5429734Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:01.5430186Z with: 2024-08-20T22:10:01.5430513Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:01.5430954Z fetch-depth: 0 2024-08-20T22:10:01.5431388Z submodules: recursive 2024-08-20T22:10:01.5431726Z quiet-checkout: true 2024-08-20T22:10:01.5432081Z repository: pytorch/pytorch 2024-08-20T22:10:01.5432566Z token: *** 2024-08-20T22:10:01.5432866Z ssh-strict: true 2024-08-20T22:10:01.5433193Z persist-credentials: true 2024-08-20T22:10:01.5433556Z clean: true 2024-08-20T22:10:01.5433902Z sparse-checkout-cone-mode: true 2024-08-20T22:10:01.5434295Z lfs: false 2024-08-20T22:10:01.5434598Z set-safe-directory: true 2024-08-20T22:10:01.5434943Z env: 2024-08-20T22:10:01.5435222Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:01.5435566Z ##[endgroup] 2024-08-20T22:10:01.6396482Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:01.6398053Z ##[group]Getting Git version info 2024-08-20T22:10:01.6398877Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:01.6399792Z [command]/usr/bin/git version 2024-08-20T22:10:01.6400166Z git version 2.40.1 2024-08-20T22:10:01.6406513Z ##[endgroup] 2024-08-20T22:10:01.6420995Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4c2230cd-2e06-4309-9c55-7bcebdc19d4f' before making global git config changes 2024-08-20T22:10:01.6422281Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:01.6426467Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:01.6476814Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:01.6480254Z ##[group]Initializing the repository 2024-08-20T22:10:01.6483008Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:01.6529082Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:01.6530248Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:01.6531127Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:01.6531709Z hint: 2024-08-20T22:10:01.6532173Z hint: git config --global init.defaultBranch 2024-08-20T22:10:01.6532657Z hint: 2024-08-20T22:10:01.6533194Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:01.6534071Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:01.6535073Z hint: 2024-08-20T22:10:01.6535430Z hint: git branch -m 2024-08-20T22:10:01.6536253Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:01.6542599Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:01.6584834Z ##[endgroup] 2024-08-20T22:10:01.6585512Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:01.6587707Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:01.6631525Z ##[endgroup] 2024-08-20T22:10:01.6632131Z ##[group]Setting up auth 2024-08-20T22:10:01.6637170Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:01.6678807Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:01.7063251Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:01.7104492Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:01.7452940Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:01.7508138Z ##[endgroup] 2024-08-20T22:10:01.7508731Z ##[group]Fetching the repository 2024-08-20T22:10:01.7514330Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:05.0172271Z remote: Enumerating objects: 1008509 2024-08-20T22:10:05.0172920Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:05.0173715Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:05.0174386Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:05.0175148Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:05.0175782Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:05.0176558Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:05.0177308Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:05.0177970Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:05.0178656Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:05.0179335Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:05.0180059Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:05.0180571Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:05.0181076Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:05.0181574Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:05.0182143Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:05.0182656Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:05.0183168Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:05.0183671Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:05.0184184Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:05.0184743Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:05.0185244Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:05.0185744Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:05.0186305Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:05.0186810Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:05.0187322Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:05.0187874Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:05.0188377Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:05.0188881Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:05.0189481Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:05.0190409Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:05.0190931Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:05.0191572Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:05.0192229Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:05.0192835Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:05.0193339Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:05.0194168Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:05.0194754Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:05.0195259Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:05.0195755Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:05.0196258Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:05.0196760Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:05.0197254Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:05.0197767Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:05.0198272Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:05.0198777Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:05.0199274Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:05.0199778Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:05.0200282Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:05.0201104Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:05.0201601Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:05.0202101Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:05.0202612Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:05.0203114Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:05.0203610Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:05.0204115Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:05.0204621Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:05.0205120Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:05.0205614Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:05.0206119Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:05.0206626Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:05.0207135Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:05.0207666Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:05.0208198Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:05.0208722Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:05.0209267Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:05.0209789Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:05.0210303Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:05.0210831Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:05.0211348Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:05.0211868Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:05.0212379Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:05.0212896Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:05.0213413Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:05.0213929Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:05.0214446Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:05.0214966Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:05.0215482Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:05.0216000Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:05.0216510Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:05.0217026Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:05.0217694Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:05.0218215Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:05.0218726Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:05.0219242Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:05.0219757Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:05.0220273Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:05.0220781Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:05.0221414Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:05.0221945Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:05.0222460Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:05.0222969Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:05.0223483Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:05.0223999Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:05.0224522Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:05.0225032Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:05.0225548Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:05.0226061Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:05.0226577Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:05.0227092Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:05.0227614Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:05.0228132Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:05.0228661Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:05.0229251Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:05.0362800Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:05.0619217Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:05.0905657Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:05.1350096Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:05.2132175Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:05.2596714Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:05.3201176Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:05.3527095Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:05.3915905Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:05.4136731Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:05.4397042Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:05.4649759Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:05.4816830Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:05.4985393Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:05.5113333Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:05.5207643Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:05.5306295Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:05.5363521Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:05.5406063Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:05.5454938Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:05.5487705Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:05.5507811Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:05.5512462Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:05.5518253Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:05.5527866Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:05.5533015Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:05.5544052Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:05.5550104Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:05.5572994Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:05.5586512Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:05.5596462Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:05.5608544Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:05.5613744Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:05.5624439Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:05.5632376Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:05.5636680Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:05.5645138Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:05.5652733Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:05.5662788Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:05.5670167Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:05.5675875Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:05.5682188Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:05.5690766Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:05.5694838Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:05.5703255Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:05.5708063Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:05.5713151Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:05.5716968Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:05.5719698Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:05.5724216Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:05.5726673Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:05.5730269Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:05.5732663Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:05.5735395Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:05.5737267Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:05.5739685Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:05.5741206Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:05.5742427Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:05.5743588Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:05.5745169Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:05.5746315Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:05.5746854Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:05.5747531Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:05.5748834Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:05.5751055Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:05.5757477Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:05.5763845Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:05.5769515Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:05.5772671Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:05.5777518Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:05.5781707Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:05.5785925Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:05.5789490Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:05.5793527Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:05.5796795Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:05.5800505Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:05.5804026Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:05.5807846Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:05.5812544Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:05.5813280Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:05.5815696Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:05.5818164Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:05.5826575Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:05.5829085Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:05.5832629Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:05.5833336Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:05.5835395Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:05.5837022Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:05.5840822Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:05.5844872Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:05.5846312Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:05.5848037Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:05.5849145Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:05.5849678Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:05.5851892Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:05.5854234Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:05.5855264Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:05.5857338Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:05.5859227Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:05.5861210Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:05.5862239Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:05.5862816Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:29.1487048Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:10:52.1245981Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:10:52.1286016Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:52.1292292Z ##[endgroup] 2024-08-20T22:10:52.1297584Z ##[group]Determining the checkout info 2024-08-20T22:10:52.1298505Z ##[endgroup] 2024-08-20T22:10:52.1299177Z ##[group]Checking out the ref 2024-08-20T22:10:52.1300147Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:53.8496935Z ##[endgroup] 2024-08-20T22:10:53.8497776Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:10:53.8503540Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:53.8570798Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:10:53.8611806Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:10:53.8652450Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:10:53.8689326Z ##[endgroup] 2024-08-20T22:10:53.8691618Z ##[group]Fetching submodules 2024-08-20T22:10:53.8693264Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:10:53.9070723Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:10:53.9438613Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:10:53.9441036Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:10:53.9445042Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:10:53.9449056Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:10:53.9454191Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:53.9458337Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:10:53.9462923Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:10:53.9467545Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:10:53.9472148Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:10:53.9477027Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:10:53.9481734Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:10:53.9486691Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:10:53.9491473Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:10:53.9496530Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:10:53.9501875Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:10:53.9508594Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:53.9517010Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:10:53.9522467Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:10:53.9527603Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:10:53.9533062Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:10:53.9538182Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:10:53.9543266Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:10:53.9548052Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:10:53.9553141Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:10:53.9558152Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:10:53.9563825Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:10:53.9568845Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:10:53.9574271Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:10:53.9579610Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:10:53.9585260Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:10:53.9590693Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:10:53.9596586Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:10:53.9602738Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:10:53.9611798Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:10:53.9646178Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:10:54.3405786Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:10:54.5231688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:10:54.7174322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:10:54.9674681Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:10:56.9218922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:08.3292405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:08.8049892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:09.2347430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:09.8293630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:11.0530072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:13.1824108Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:19.1982412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:20.6115266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:22.3855185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:23.6729183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:24.0941112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:24.4189724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:25.5596008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:25.9889449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:26.2656212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:27.9144938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:28.6473744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:29.3088770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:35.4944671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:37.4373671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:41.6471883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:41.8920192Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:11:51.0761017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:11:51.2614900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:11:51.4669102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:11:52.3822550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:11:52.6842964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:11:53.3037162Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:11:53.7448357Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:11:53.7599917Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:11:53.7718616Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:11:53.8025473Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:11:53.8464195Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:11:54.9769928Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:11:55.0055543Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:11:55.0586484Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:11:55.1691025Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:11:55.2091801Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:11:55.8277009Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:11:56.1128459Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:11:56.2067849Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:11:56.2090119Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:11:56.2093585Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:11:56.2096979Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:11:56.2101906Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:11:56.2104408Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:11:56.2139575Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:11:57.3152292Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:11:57.9833442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:00.0771198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:01.1350912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:01.4825459Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:01.5930481Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:02.0986663Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:02.1697772Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:02.1847280Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:02.3363494Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:02.3827312Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:02.4286148Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:02.4608598Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:02.5141419Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:02.5297757Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:02.5316969Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:02.5351741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:15.8613630Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:15.8833795Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:15.9835255Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:15.9859199Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:15.9862224Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:15.9865847Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:15.9897932Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:16.5091890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:17.7355961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:18.9012082Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:18.9033468Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:18.9036948Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:18.9041176Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:18.9045082Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:18.9048833Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:18.9052765Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:18.9056621Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:18.9060525Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:18.9092303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:19.7512276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:20.2144303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:21.3980919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:21.6896341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:22.2099248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:23.2698415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:30.5635827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:30.9684591Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:30.9912912Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:31.0342125Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:31.0513576Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:31.0533871Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:31.0565179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:31.3525850Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:31.3754111Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:31.4234745Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:31.5460833Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:31.5667384Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:31.6116507Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:31.6777092Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:31.7227540Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:31.7524763Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:31.8782413Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:32.3903542Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:32.3938529Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:32.3941523Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:32.3973290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:32.8348412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:33.8235477Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:33.8645602Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:33.9540074Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:33.9561383Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:33.9564953Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:33.9568338Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:33.9571834Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:33.9575780Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:33.9578922Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:33.9582870Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:33.9586419Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:33.9619242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:34.3791830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:35.4873061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:35.8508375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:42.2133363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:42.4862186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:42.6920800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:42.9910419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:12:49.7552852Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:12:49.8024849Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:12:49.8213720Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:12:49.9444717Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:12:49.9604892Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:12:49.9793006Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:12:49.9993343Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:12:50.0012558Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:50.0015928Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:50.0046886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:12:51.9096496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:12:53.3071980Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:12:53.3601116Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:54.0260646Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:12:54.0405989Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:12:54.3582334Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:12:54.3605450Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:54.3608730Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:54.3639730Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:12:54.8630375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:12:55.9339529Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:12:56.0156228Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:12:56.0275076Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:12:56.0428777Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:12:56.0875862Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:12:56.1216597Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:12:56.1705956Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:12:56.2046307Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:12:56.2064215Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:56.2067614Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:56.2071225Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:56.2074841Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:56.2107145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:12:57.2341387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:12:57.4652062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:12:58.6196023Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:12:59.5713581Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:12:59.5909104Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:12:59.6704200Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:12:59.7040555Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:12:59.7060058Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:59.7094462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:12:59.8972932Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:12:59.9023078Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:12:59.9393782Z Entering 'android/libs/fbjni' 2024-08-20T22:12:59.9448547Z Entering 'third_party/FP16' 2024-08-20T22:12:59.9501079Z Entering 'third_party/FXdiv' 2024-08-20T22:12:59.9553611Z Entering 'third_party/NNPACK' 2024-08-20T22:12:59.9606172Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:59.9657132Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:59.9726111Z Entering 'third_party/benchmark' 2024-08-20T22:12:59.9778142Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:59.9835077Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:59.9887256Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:59.9939566Z Entering 'third_party/cutlass' 2024-08-20T22:12:59.9998950Z Entering 'third_party/eigen' 2024-08-20T22:13:00.0057423Z Entering 'third_party/fbgemm' 2024-08-20T22:13:00.0110302Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:00.0160592Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:00.0211361Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:00.0268046Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:00.0317757Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:00.0370167Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:00.0426375Z Entering 'third_party/fmt' 2024-08-20T22:13:00.0478686Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:00.0532371Z Entering 'third_party/gloo' 2024-08-20T22:13:00.0584928Z Entering 'third_party/googletest' 2024-08-20T22:13:00.0637714Z Entering 'third_party/ideep' 2024-08-20T22:13:00.0688144Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:00.0749691Z Entering 'third_party/ittapi' 2024-08-20T22:13:00.0802131Z Entering 'third_party/kineto' 2024-08-20T22:13:00.0855745Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:00.0905739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:00.0958310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:00.1010074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:00.1060592Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:00.1110006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:00.1164540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:00.1215831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:00.1266728Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:00.1318702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:00.1373380Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:00.1423690Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:00.1477133Z Entering 'third_party/mimalloc' 2024-08-20T22:13:00.1532180Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:00.1585135Z Entering 'third_party/nlohmann' 2024-08-20T22:13:00.1640729Z Entering 'third_party/onnx' 2024-08-20T22:13:00.1706241Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:00.1762335Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:00.1821708Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:00.1874881Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:00.1924817Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:00.1974431Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:00.2025754Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:00.2076705Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:00.2126469Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:00.2175897Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:00.2225208Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:00.2277488Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:00.2333255Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:00.2405771Z Entering 'third_party/pocketfft' 2024-08-20T22:13:00.2459548Z Entering 'third_party/protobuf' 2024-08-20T22:13:00.2513245Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:00.2563035Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:00.2617479Z Entering 'third_party/psimd' 2024-08-20T22:13:00.2671246Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:00.2722777Z Entering 'third_party/pybind11' 2024-08-20T22:13:00.2774594Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:00.2827196Z Entering 'third_party/sleef' 2024-08-20T22:13:00.2879886Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:00.2931755Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:00.2982242Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:00.3032411Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:00.3083308Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:00.3132999Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:00.3207771Z ##[endgroup] 2024-08-20T22:13:00.3211131Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:00.3212820Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:00.3576201Z Entering 'android/libs/fbjni' 2024-08-20T22:13:00.3645587Z Entering 'third_party/FP16' 2024-08-20T22:13:00.3716191Z Entering 'third_party/FXdiv' 2024-08-20T22:13:00.3785077Z Entering 'third_party/NNPACK' 2024-08-20T22:13:00.3855777Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:00.3924996Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:00.4010820Z Entering 'third_party/benchmark' 2024-08-20T22:13:00.4079511Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:00.4150640Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:00.4220290Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:00.4289906Z Entering 'third_party/cutlass' 2024-08-20T22:13:00.4367791Z Entering 'third_party/eigen' 2024-08-20T22:13:00.4443051Z Entering 'third_party/fbgemm' 2024-08-20T22:13:00.4510961Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:00.4576726Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:00.4648914Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:00.4720753Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:00.4790991Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:00.4864684Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:00.4936845Z Entering 'third_party/fmt' 2024-08-20T22:13:00.5006663Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:00.5076314Z Entering 'third_party/gloo' 2024-08-20T22:13:00.5146273Z Entering 'third_party/googletest' 2024-08-20T22:13:00.5215887Z Entering 'third_party/ideep' 2024-08-20T22:13:00.5282921Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:00.5360191Z Entering 'third_party/ittapi' 2024-08-20T22:13:00.5429153Z Entering 'third_party/kineto' 2024-08-20T22:13:00.5495950Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:00.5561221Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:00.5630134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:00.5696608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:00.5763200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:00.5827155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:00.5898115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:00.5965220Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:00.6031875Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:00.6098594Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:00.6168640Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:00.6235310Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:00.6305111Z Entering 'third_party/mimalloc' 2024-08-20T22:13:00.6374814Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:00.6444794Z Entering 'third_party/nlohmann' 2024-08-20T22:13:00.6513909Z Entering 'third_party/onnx' 2024-08-20T22:13:00.6595223Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:00.6663559Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:00.6734701Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:00.6803682Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:00.6869027Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:00.6939021Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:00.7004451Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:00.7074223Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:00.7140823Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:00.7206946Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:00.7271423Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:00.7343870Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:00.7413429Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:00.7499369Z Entering 'third_party/pocketfft' 2024-08-20T22:13:00.7569733Z Entering 'third_party/protobuf' 2024-08-20T22:13:00.7640213Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:00.7705737Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:00.7775948Z Entering 'third_party/psimd' 2024-08-20T22:13:00.7844618Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:00.7914257Z Entering 'third_party/pybind11' 2024-08-20T22:13:00.7982344Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:00.8052656Z Entering 'third_party/sleef' 2024-08-20T22:13:00.8121217Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:00.8188585Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:00.8254366Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:00.8319701Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:00.8389307Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:00.8456805Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:00.8547566Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:00.8909762Z Entering 'android/libs/fbjni' 2024-08-20T22:13:00.8969234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:00.8990436Z Entering 'third_party/FP16' 2024-08-20T22:13:00.9051698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:00.9072798Z Entering 'third_party/FXdiv' 2024-08-20T22:13:00.9134175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:00.9155377Z Entering 'third_party/NNPACK' 2024-08-20T22:13:00.9220175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:00.9242770Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:00.9305228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:00.9327068Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:00.9388333Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:00.9426161Z Entering 'third_party/benchmark' 2024-08-20T22:13:00.9487710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:00.9511166Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:00.9572654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:00.9594073Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:00.9659268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:00.9681589Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:00.9744881Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:00.9766996Z Entering 'third_party/cutlass' 2024-08-20T22:13:00.9831340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:00.9861086Z Entering 'third_party/eigen' 2024-08-20T22:13:00.9925671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:00.9950340Z Entering 'third_party/fbgemm' 2024-08-20T22:13:01.0019169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:01.0039432Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:01.0101075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:01.0123159Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:01.0183676Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:01.0205681Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:01.0266563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:01.0292700Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:01.0352565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:01.0372307Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:01.0433621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:01.0455874Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:01.0518425Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:01.0543195Z Entering 'third_party/fmt' 2024-08-20T22:13:01.0607651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:01.0629735Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:01.0696576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:01.0719181Z Entering 'third_party/gloo' 2024-08-20T22:13:01.0781813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:01.0804084Z Entering 'third_party/googletest' 2024-08-20T22:13:01.0865568Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:01.0887758Z Entering 'third_party/ideep' 2024-08-20T22:13:01.0950838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:01.0970513Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:01.1031356Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:01.1059770Z Entering 'third_party/ittapi' 2024-08-20T22:13:01.1123070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:01.1144649Z Entering 'third_party/kineto' 2024-08-20T22:13:01.1209357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:01.1229495Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:01.1296829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:01.1315455Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:01.1383185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:01.1405581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:01.1467755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:01.1488554Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:01.1555026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:01.1576502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:01.1638335Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:01.1656392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:01.1720787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:01.1747028Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:01.1810125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:01.1830430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:01.1892782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:01.1913616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:01.1975652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:01.1996850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:01.2060960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:01.2085939Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:01.2147848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:01.2167915Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:01.2229997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:01.2253002Z Entering 'third_party/mimalloc' 2024-08-20T22:13:01.2315944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:01.2337928Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:01.2402083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:01.2422862Z Entering 'third_party/nlohmann' 2024-08-20T22:13:01.2484728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:01.2507895Z Entering 'third_party/onnx' 2024-08-20T22:13:01.2575114Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:01.2611252Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:01.2673641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:01.2696004Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:01.2759143Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:01.2782836Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:01.2845805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:01.2867294Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:01.2933856Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:01.2953722Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:01.3019158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:01.3038798Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:01.3104470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:01.3124061Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:01.3188992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:01.3210493Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:01.3271256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:01.3290471Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:01.3351973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:01.3372174Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:01.3440192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:01.3458025Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:01.3520055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:01.3541978Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:01.3604201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:01.3626803Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:01.3687589Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:01.3729665Z Entering 'third_party/pocketfft' 2024-08-20T22:13:01.3791922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:01.3813941Z Entering 'third_party/protobuf' 2024-08-20T22:13:01.3881140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:01.3905864Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:01.3968073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:01.3988275Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:01.4049827Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:01.4076288Z Entering 'third_party/psimd' 2024-08-20T22:13:01.4140728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:01.4161242Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:01.4224664Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:01.4246327Z Entering 'third_party/pybind11' 2024-08-20T22:13:01.4310502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:01.4333014Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:01.4394653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:01.4416793Z Entering 'third_party/sleef' 2024-08-20T22:13:01.4479221Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:01.4500207Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:01.4563753Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:01.4583752Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:01.4646573Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:01.4666755Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:01.4733703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:01.4753387Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:01.4814543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:01.4834524Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:01.4899019Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:01.4921660Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:01.4983559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:01.5950677Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:01.6318954Z Entering 'android/libs/fbjni' 2024-08-20T22:13:01.6372088Z Entering 'third_party/FP16' 2024-08-20T22:13:01.6425224Z Entering 'third_party/FXdiv' 2024-08-20T22:13:01.6477535Z Entering 'third_party/NNPACK' 2024-08-20T22:13:01.6530537Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:01.6582951Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:01.6651262Z Entering 'third_party/benchmark' 2024-08-20T22:13:01.6704699Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:01.6756205Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:01.6810228Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:01.6863104Z Entering 'third_party/cutlass' 2024-08-20T22:13:01.6923342Z Entering 'third_party/eigen' 2024-08-20T22:13:01.6976892Z Entering 'third_party/fbgemm' 2024-08-20T22:13:01.7030327Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:01.7080203Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:01.7131511Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:01.7188076Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:01.7238779Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:01.7291577Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:01.7348257Z Entering 'third_party/fmt' 2024-08-20T22:13:01.7400936Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:01.7454684Z Entering 'third_party/gloo' 2024-08-20T22:13:01.7507646Z Entering 'third_party/googletest' 2024-08-20T22:13:01.7560380Z Entering 'third_party/ideep' 2024-08-20T22:13:01.7613092Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:01.7675885Z Entering 'third_party/ittapi' 2024-08-20T22:13:01.7730489Z Entering 'third_party/kineto' 2024-08-20T22:13:01.7783028Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:01.7838018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:01.7890859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:01.7942324Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:01.7993368Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:01.8042486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:01.8102348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:01.8153220Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:01.8205349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:01.8256941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:01.8310846Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:01.8364906Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:01.8419014Z Entering 'third_party/mimalloc' 2024-08-20T22:13:01.8471909Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:01.8528531Z Entering 'third_party/nlohmann' 2024-08-20T22:13:01.8583386Z Entering 'third_party/onnx' 2024-08-20T22:13:01.8649657Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:01.8701491Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:01.8757394Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:01.8810991Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:01.8861026Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:01.8910395Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:01.8959706Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:01.9014783Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:01.9064598Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:01.9114213Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:01.9163144Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:01.9218337Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:01.9271729Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:01.9346418Z Entering 'third_party/pocketfft' 2024-08-20T22:13:01.9398847Z Entering 'third_party/protobuf' 2024-08-20T22:13:01.9459077Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:01.9514227Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:01.9570104Z Entering 'third_party/psimd' 2024-08-20T22:13:01.9624523Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:01.9676543Z Entering 'third_party/pybind11' 2024-08-20T22:13:01.9729379Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:01.9781428Z Entering 'third_party/sleef' 2024-08-20T22:13:01.9834226Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:01.9885497Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:01.9936514Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:01.9985865Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:02.0036120Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:02.0085769Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:02.0162154Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:02.0526084Z Entering 'android/libs/fbjni' 2024-08-20T22:13:02.0576964Z Entering 'third_party/FP16' 2024-08-20T22:13:02.0629398Z Entering 'third_party/FXdiv' 2024-08-20T22:13:02.0682792Z Entering 'third_party/NNPACK' 2024-08-20T22:13:02.0735211Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:02.0787084Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:02.0855943Z Entering 'third_party/benchmark' 2024-08-20T22:13:02.0910390Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:02.0963498Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:02.1016619Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:02.1068768Z Entering 'third_party/cutlass' 2024-08-20T22:13:02.1130913Z Entering 'third_party/eigen' 2024-08-20T22:13:02.1185152Z Entering 'third_party/fbgemm' 2024-08-20T22:13:02.1237179Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:02.1287087Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:02.1338247Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:02.1394361Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:02.1446072Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:02.1498686Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:02.1555539Z Entering 'third_party/fmt' 2024-08-20T22:13:02.1607892Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:02.1660072Z Entering 'third_party/gloo' 2024-08-20T22:13:02.1712439Z Entering 'third_party/googletest' 2024-08-20T22:13:02.1764748Z Entering 'third_party/ideep' 2024-08-20T22:13:02.1814940Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:02.1875608Z Entering 'third_party/ittapi' 2024-08-20T22:13:02.1930703Z Entering 'third_party/kineto' 2024-08-20T22:13:02.1981400Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:02.2030300Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:02.2082255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:02.2133751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:02.2184006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:02.2232689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:02.2287302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:02.2337630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:02.2387465Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:02.2438990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:02.2492126Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:02.2544041Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:02.2597787Z Entering 'third_party/mimalloc' 2024-08-20T22:13:02.2654621Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:02.2706642Z Entering 'third_party/nlohmann' 2024-08-20T22:13:02.2759107Z Entering 'third_party/onnx' 2024-08-20T22:13:02.2824421Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:02.2875196Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:02.2931950Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:02.2984552Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:02.3036745Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:02.3086283Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:02.3136595Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:02.3187454Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:02.3242997Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:02.3292557Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:02.3341358Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:02.3394712Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:02.3447441Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:02.3525343Z Entering 'third_party/pocketfft' 2024-08-20T22:13:02.3579119Z Entering 'third_party/protobuf' 2024-08-20T22:13:02.3697115Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:02.3748046Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:02.3804955Z Entering 'third_party/psimd' 2024-08-20T22:13:02.3857795Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:02.3910378Z Entering 'third_party/pybind11' 2024-08-20T22:13:02.3963435Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:02.4015458Z Entering 'third_party/sleef' 2024-08-20T22:13:02.4067847Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:02.4119557Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:02.4169428Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:02.4220590Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:02.4270440Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:02.4325326Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:02.4398525Z ##[endgroup] 2024-08-20T22:13:02.4447592Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:02.4484122Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:02.4709692Z Prepare all required actions 2024-08-20T22:13:02.4710159Z Getting action download info 2024-08-20T22:13:02.6022916Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:02.6023342Z env: 2024-08-20T22:13:02.6023618Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.6023956Z ##[endgroup] 2024-08-20T22:13:02.6082394Z ##[group]Run set -euo pipefail 2024-08-20T22:13:02.6082827Z set -euo pipefail 2024-08-20T22:13:02.6083231Z function get_ec2_metadata() { 2024-08-20T22:13:02.6083787Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:02.6084698Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:02.6085471Z  category=$1 2024-08-20T22:13:02.6086036Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:02.6086689Z  runner_name_str=i-0284de2732bb465d7 2024-08-20T22:13:02.6087189Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:02.6087699Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:02.6088276Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:02.6088980Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:02.6089593Z  else 2024-08-20T22:13:02.6090101Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:02.6090682Z  fi 2024-08-20T22:13:02.6090976Z } 2024-08-20T22:13:02.6091340Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:02.6091943Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:02.6092614Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:02.6093202Z echo "system info $(uname -a)" 2024-08-20T22:13:02.6105776Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.6106290Z env: 2024-08-20T22:13:02.6106580Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.6106934Z ##[endgroup] 2024-08-20T22:13:02.6215546Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:02.6273120Z instance-id: i-0284de2732bb465d7 2024-08-20T22:13:02.6329890Z instance-type: g5.4xlarge 2024-08-20T22:13:02.6344688Z system info Linux ip-10-0-11-223.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:02.6367808Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:02.6368738Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:02.6377623Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.6378288Z env: 2024-08-20T22:13:02.6378573Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.6378925Z ##[endgroup] 2024-08-20T22:13:02.6452746Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:02.6453359Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:02.6453905Z  echo "Docker daemon is running..."; 2024-08-20T22:13:02.6454360Z else 2024-08-20T22:13:02.6454855Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:02.6455439Z fi 2024-08-20T22:13:02.6463487Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.6463988Z env: 2024-08-20T22:13:02.6464273Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.6464625Z ##[endgroup] 2024-08-20T22:13:02.6555871Z Docker daemon is running... 2024-08-20T22:13:02.6629085Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:02.6629648Z with: 2024-08-20T22:13:02.6629957Z shell: bash 2024-08-20T22:13:02.6630279Z timeout_minutes: 5 2024-08-20T22:13:02.6630612Z max_attempts: 3 2024-08-20T22:13:02.6630934Z retry_wait_seconds: 30 2024-08-20T22:13:02.6634123Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:02.6637169Z polling_interval_seconds: 1 2024-08-20T22:13:02.6637550Z warning_on_retry: true 2024-08-20T22:13:02.6637913Z continue_on_error: false 2024-08-20T22:13:02.6638265Z env: 2024-08-20T22:13:02.6638544Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.6638904Z AWS_RETRY_MODE: standard 2024-08-20T22:13:02.6639264Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:02.6639616Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:02.6640021Z ##[endgroup] 2024-08-20T22:13:03.8295458Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:03.8296314Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:03.8297174Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:03.8297699Z 2024-08-20T22:13:03.8297825Z Login Succeeded 2024-08-20T22:13:04.7201563Z Command completed after 1 attempt(s). 2024-08-20T22:13:04.7265178Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:04.7265903Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:04.7266570Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:04.7277150Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:04.7277661Z env: 2024-08-20T22:13:04.7277949Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:04.7278303Z ##[endgroup] 2024-08-20T22:13:04.7377951Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:04.7378738Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:04.7379346Z # shellcheck disable=SC2046 2024-08-20T22:13:04.7379820Z docker stop $(docker ps -q) || true 2024-08-20T22:13:04.7380308Z # Prune all of the docker images 2024-08-20T22:13:04.7380771Z docker system prune -af 2024-08-20T22:13:04.7390070Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:04.7390579Z env: 2024-08-20T22:13:04.7390859Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:04.7391211Z ##[endgroup] 2024-08-20T22:13:04.7710597Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:04.7711715Z See 'docker stop --help'. 2024-08-20T22:13:04.7711975Z 2024-08-20T22:13:04.7712202Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:04.7712584Z 2024-08-20T22:13:04.7712741Z Stop one or more running containers 2024-08-20T22:13:04.7911788Z Total reclaimed space: 0B 2024-08-20T22:13:04.7957382Z ##[group]Run set +e 2024-08-20T22:13:04.7957751Z set +e 2024-08-20T22:13:04.7958063Z set -x 2024-08-20T22:13:04.7958361Z  2024-08-20T22:13:04.7958686Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:04.7959493Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:04.7960739Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:04.7961500Z # one is returned at random 2024-08-20T22:13:04.7962055Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:04.7962568Z  2024-08-20T22:13:04.7962907Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:04.7963537Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:04.7964513Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:04.7965075Z  2024-08-20T22:13:04.7965412Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:04.7965972Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:04.7966491Z  exit 1 2024-08-20T22:13:04.7966805Z  fi 2024-08-20T22:13:04.7967097Z fi 2024-08-20T22:13:04.7967376Z  2024-08-20T22:13:04.7967749Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:04.7968275Z  # Clean up any old records first 2024-08-20T22:13:04.7968796Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:04.7969258Z fi 2024-08-20T22:13:04.7969535Z  2024-08-20T22:13:04.7969981Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:04.7970535Z cat /etc/hosts 2024-08-20T22:13:04.7979460Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:04.7979963Z env: 2024-08-20T22:13:04.7980257Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:04.7980613Z ##[endgroup] 2024-08-20T22:13:04.8009089Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:04.8015804Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:04.8016664Z ++ tail -n1 2024-08-20T22:13:04.8473610Z + RESOLVED_IP=18.160.10.36 2024-08-20T22:13:04.8474067Z + '[' -z 18.160.10.36 ']' 2024-08-20T22:13:04.8474512Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:04.8491807Z + echo '18.160.10.36 download.pytorch.org' 2024-08-20T22:13:04.8492366Z + sudo tee -a /etc/hosts 2024-08-20T22:13:05.0994065Z 18.160.10.36 download.pytorch.org 2024-08-20T22:13:05.1016418Z + cat /etc/hosts 2024-08-20T22:13:05.1027406Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:05.1039701Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:05.1040496Z 18.160.10.36 download.pytorch.org 2024-08-20T22:13:05.1194639Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:05.1195257Z with: 2024-08-20T22:13:05.1196381Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1197626Z docker-build-dir: .ci/docker 2024-08-20T22:13:05.1198021Z working-directory: . 2024-08-20T22:13:05.1198499Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.1199031Z force-push: false 2024-08-20T22:13:05.1199340Z env: 2024-08-20T22:13:05.1199622Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:05.1199967Z ##[endgroup] 2024-08-20T22:13:05.1219596Z ##[group]Run set -ex 2024-08-20T22:13:05.1219964Z set -ex 2024-08-20T22:13:05.1220258Z  2024-08-20T22:13:05.1220999Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:05.1222056Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:05.1222869Z # job could then download the pre-built image as usual 2024-08-20T22:13:05.1223626Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:05.1224313Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1224969Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1225542Z  2024-08-20T22:13:05.1226069Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:05.1226707Z  exit 0 2024-08-20T22:13:05.1227016Z else 2024-08-20T22:13:05.1227387Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1227855Z fi 2024-08-20T22:13:05.1228138Z  2024-08-20T22:13:05.1228627Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:05.1229494Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:05.1230292Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:05.1231016Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:05.1231756Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1232472Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1233037Z else 2024-08-20T22:13:05.1233482Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:05.1234168Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1235089Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1235853Z fi 2024-08-20T22:13:05.1246935Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:05.1247442Z env: 2024-08-20T22:13:05.1247729Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:05.1248088Z REPO_NAME: pytorch 2024-08-20T22:13:05.1249253Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1250492Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:05.1251001Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.1251563Z ##[endgroup] 2024-08-20T22:13:05.1282381Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:05.1283127Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:05.1283725Z + echo skip=false 2024-08-20T22:13:05.1285888Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:05.1290308Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1291894Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:05.1315986Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1317072Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1318965Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1351951Z ##[group]Run set +e 2024-08-20T22:13:05.1352407Z set +e 2024-08-20T22:13:05.1352892Z set -x 2024-08-20T22:13:05.1353280Z  2024-08-20T22:13:05.1353793Z login() { 2024-08-20T22:13:05.1354620Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:05.1355442Z } 2024-08-20T22:13:05.1355836Z  2024-08-20T22:13:05.1356220Z retry () { 2024-08-20T22:13:05.1356710Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:05.1357279Z } 2024-08-20T22:13:05.1357692Z  2024-08-20T22:13:05.1358092Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:05.1358608Z  2024-08-20T22:13:05.1359240Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:05.1360075Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:05.1360739Z  exit 0 2024-08-20T22:13:05.1361191Z fi 2024-08-20T22:13:05.1361588Z  2024-08-20T22:13:05.1362232Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:05.1363140Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:05.1364009Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:05.1364840Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:05.1365565Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:05.1366063Z else 2024-08-20T22:13:05.1366683Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:05.1367506Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:05.1368114Z fi 2024-08-20T22:13:05.1368500Z  2024-08-20T22:13:05.1368937Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:05.1369543Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1370183Z  2024-08-20T22:13:05.1370905Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:05.1371769Z  exit 0 2024-08-20T22:13:05.1372234Z fi 2024-08-20T22:13:05.1372603Z  2024-08-20T22:13:05.1373107Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:05.1374266Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:05.1375176Z  exit 1 2024-08-20T22:13:05.1375591Z fi 2024-08-20T22:13:05.1375993Z  2024-08-20T22:13:05.1376585Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:05.1377675Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:05.1378700Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:05.1379738Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:05.1380992Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:05.1381999Z fi 2024-08-20T22:13:05.1382407Z  2024-08-20T22:13:05.1382826Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:05.1391334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:05.1392003Z env: 2024-08-20T22:13:05.1392382Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:05.1392836Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:05.1393454Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:05.1394860Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1396211Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.1396955Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.1397642Z ##[endgroup] 2024-08-20T22:13:05.1427034Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.1427833Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.1429925Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:05.1431289Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.6746200Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:05.6747416Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:05.6748536Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:05.6749096Z 2024-08-20T22:13:05.6749360Z Login Succeeded 2024-08-20T22:13:05.6771533Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.8984815Z { 2024-08-20T22:13:05.8985269Z "schemaVersion": 2, 2024-08-20T22:13:05.8986372Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:05.8987378Z "config": { 2024-08-20T22:13:05.8988277Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:05.8988977Z "size": 48711, 2024-08-20T22:13:05.8989953Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:05.8991378Z }, 2024-08-20T22:13:05.8993007Z "layers": [ 2024-08-20T22:13:05.8993675Z { 2024-08-20T22:13:05.8994538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.8995631Z "size": 28580681, 2024-08-20T22:13:05.8996473Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:05.8997293Z }, 2024-08-20T22:13:05.8997699Z { 2024-08-20T22:13:05.8998490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.8999286Z "size": 7943451, 2024-08-20T22:13:05.9001225Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:05.9001957Z }, 2024-08-20T22:13:05.9002567Z { 2024-08-20T22:13:05.9003591Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9004766Z "size": 55728572, 2024-08-20T22:13:05.9005998Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:05.9007021Z }, 2024-08-20T22:13:05.9007786Z { 2024-08-20T22:13:05.9008449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9009193Z "size": 186, 2024-08-20T22:13:05.9010687Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:05.9011799Z }, 2024-08-20T22:13:05.9012333Z { 2024-08-20T22:13:05.9013041Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9013979Z "size": 6886, 2024-08-20T22:13:05.9015107Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:05.9016051Z }, 2024-08-20T22:13:05.9016930Z { 2024-08-20T22:13:05.9017741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9018737Z "size": 1329001756, 2024-08-20T22:13:05.9019655Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:05.9020739Z }, 2024-08-20T22:13:05.9021167Z { 2024-08-20T22:13:05.9021919Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9022648Z "size": 62501, 2024-08-20T22:13:05.9023313Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:05.9024037Z }, 2024-08-20T22:13:05.9024407Z { 2024-08-20T22:13:05.9024960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9025622Z "size": 1684, 2024-08-20T22:13:05.9026257Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:05.9027164Z }, 2024-08-20T22:13:05.9027556Z { 2024-08-20T22:13:05.9028083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9028739Z "size": 1523, 2024-08-20T22:13:05.9029456Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:05.9030204Z }, 2024-08-20T22:13:05.9030509Z { 2024-08-20T22:13:05.9031135Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9031799Z "size": 2528295403, 2024-08-20T22:13:05.9032501Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:05.9033189Z }, 2024-08-20T22:13:05.9033528Z { 2024-08-20T22:13:05.9034111Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9034751Z "size": 86016, 2024-08-20T22:13:05.9035399Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:05.9036181Z }, 2024-08-20T22:13:05.9036546Z { 2024-08-20T22:13:05.9037048Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9037803Z "size": 1822, 2024-08-20T22:13:05.9038442Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:05.9039154Z }, 2024-08-20T22:13:05.9039531Z { 2024-08-20T22:13:05.9040057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9040887Z "size": 246761563, 2024-08-20T22:13:05.9041658Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:05.9042394Z }, 2024-08-20T22:13:05.9042734Z { 2024-08-20T22:13:05.9043346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9043968Z "size": 544, 2024-08-20T22:13:05.9044612Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:05.9045392Z }, 2024-08-20T22:13:05.9045734Z { 2024-08-20T22:13:05.9046252Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9046958Z "size": 1258, 2024-08-20T22:13:05.9047587Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:05.9048306Z }, 2024-08-20T22:13:05.9048655Z { 2024-08-20T22:13:05.9049184Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9049862Z "size": 484, 2024-08-20T22:13:05.9050488Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:05.9051208Z }, 2024-08-20T22:13:05.9051577Z { 2024-08-20T22:13:05.9052157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9052800Z "size": 91711766, 2024-08-20T22:13:05.9053458Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:05.9054209Z }, 2024-08-20T22:13:05.9054578Z { 2024-08-20T22:13:05.9055075Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9055776Z "size": 3390, 2024-08-20T22:13:05.9056538Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:05.9057288Z }, 2024-08-20T22:13:05.9057619Z { 2024-08-20T22:13:05.9058158Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9058896Z "size": 1909, 2024-08-20T22:13:05.9059511Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:05.9060217Z }, 2024-08-20T22:13:05.9060629Z { 2024-08-20T22:13:05.9061162Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9061781Z "size": 701, 2024-08-20T22:13:05.9062507Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:05.9063237Z }, 2024-08-20T22:13:05.9063589Z { 2024-08-20T22:13:05.9064268Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9064940Z "size": 2864254879, 2024-08-20T22:13:05.9065680Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:05.9066424Z }, 2024-08-20T22:13:05.9066780Z { 2024-08-20T22:13:05.9067332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9068029Z "size": 380, 2024-08-20T22:13:05.9068627Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:05.9069366Z }, 2024-08-20T22:13:05.9069758Z { 2024-08-20T22:13:05.9070318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9070944Z "size": 12900, 2024-08-20T22:13:05.9071644Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:05.9072414Z }, 2024-08-20T22:13:05.9072756Z { 2024-08-20T22:13:05.9073291Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9073974Z "size": 803, 2024-08-20T22:13:05.9074606Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:05.9075317Z }, 2024-08-20T22:13:05.9075735Z { 2024-08-20T22:13:05.9076232Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9076957Z "size": 106, 2024-08-20T22:13:05.9077586Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:05.9078333Z }, 2024-08-20T22:13:05.9078661Z { 2024-08-20T22:13:05.9079224Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9079925Z "size": 503, 2024-08-20T22:13:05.9080715Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:05.9081378Z }, 2024-08-20T22:13:05.9081764Z { 2024-08-20T22:13:05.9082366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9082990Z "size": 121477267, 2024-08-20T22:13:05.9083707Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:05.9084409Z }, 2024-08-20T22:13:05.9084758Z { 2024-08-20T22:13:05.9085327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9086011Z "size": 109, 2024-08-20T22:13:05.9086643Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:05.9087419Z }, 2024-08-20T22:13:05.9087721Z { 2024-08-20T22:13:05.9088258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9088996Z "size": 490, 2024-08-20T22:13:05.9089619Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:05.9090304Z }, 2024-08-20T22:13:05.9090703Z { 2024-08-20T22:13:05.9091238Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9091878Z "size": 297, 2024-08-20T22:13:05.9092558Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:05.9093281Z }, 2024-08-20T22:13:05.9094881Z { 2024-08-20T22:13:05.9095451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9096127Z "size": 103, 2024-08-20T22:13:05.9096775Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:05.9097542Z }, 2024-08-20T22:13:05.9097866Z { 2024-08-20T22:13:05.9098402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9099099Z "size": 1473, 2024-08-20T22:13:05.9099721Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:05.9100821Z }, 2024-08-20T22:13:05.9101283Z { 2024-08-20T22:13:05.9101815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9102435Z "size": 594729516, 2024-08-20T22:13:05.9103159Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:05.9104023Z }, 2024-08-20T22:13:05.9104454Z { 2024-08-20T22:13:05.9104959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9105619Z "size": 164, 2024-08-20T22:13:05.9106325Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:05.9107026Z }, 2024-08-20T22:13:05.9107329Z { 2024-08-20T22:13:05.9107960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9108615Z "size": 564, 2024-08-20T22:13:05.9109209Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:05.9109979Z }, 2024-08-20T22:13:05.9110321Z { 2024-08-20T22:13:05.9110877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9111540Z "size": 35874203, 2024-08-20T22:13:05.9112212Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:05.9112966Z }, 2024-08-20T22:13:05.9113355Z { 2024-08-20T22:13:05.9113854Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9114538Z "size": 104, 2024-08-20T22:13:05.9115245Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:05.9115908Z }, 2024-08-20T22:13:05.9116268Z { 2024-08-20T22:13:05.9116915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9117591Z "size": 425, 2024-08-20T22:13:05.9118207Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:05.9118975Z }, 2024-08-20T22:13:05.9119336Z { 2024-08-20T22:13:05.9119868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9120660Z "size": 20262183, 2024-08-20T22:13:05.9121335Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:05.9122074Z }, 2024-08-20T22:13:05.9122491Z { 2024-08-20T22:13:05.9122984Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9123643Z "size": 438, 2024-08-20T22:13:05.9124352Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:05.9125025Z }, 2024-08-20T22:13:05.9125360Z { 2024-08-20T22:13:05.9125950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9126595Z "size": 701, 2024-08-20T22:13:05.9127200Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:05.9127972Z }, 2024-08-20T22:13:05.9128342Z { 2024-08-20T22:13:05.9128938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9129550Z "size": 141, 2024-08-20T22:13:05.9130185Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:05.9131055Z + exit 0 2024-08-20T22:13:05.9131417Z }, 2024-08-20T22:13:05.9131788Z { 2024-08-20T22:13:05.9132376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9133183Z "size": 135, 2024-08-20T22:13:05.9133843Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:05.9134603Z }, 2024-08-20T22:13:05.9134946Z { 2024-08-20T22:13:05.9135510Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9136216Z "size": 4838068375, 2024-08-20T22:13:05.9136900Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:05.9137570Z }, 2024-08-20T22:13:05.9137972Z { 2024-08-20T22:13:05.9138532Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9139193Z "size": 194, 2024-08-20T22:13:05.9139844Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:05.9140578Z }, 2024-08-20T22:13:05.9141017Z { 2024-08-20T22:13:05.9141575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9142296Z "size": 564, 2024-08-20T22:13:05.9142939Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:05.9143720Z }, 2024-08-20T22:13:05.9144025Z { 2024-08-20T22:13:05.9144558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9145281Z "size": 43162921, 2024-08-20T22:13:05.9145925Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:05.9146599Z }, 2024-08-20T22:13:05.9147010Z { 2024-08-20T22:13:05.9147545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9148161Z "size": 106, 2024-08-20T22:13:05.9148906Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:05.9149613Z }, 2024-08-20T22:13:05.9150026Z { 2024-08-20T22:13:05.9150524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9151188Z "size": 1380, 2024-08-20T22:13:05.9151884Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:05.9152580Z }, 2024-08-20T22:13:05.9152886Z { 2024-08-20T22:13:05.9153486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9154144Z "size": 701, 2024-08-20T22:13:05.9154803Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:05.9155515Z }, 2024-08-20T22:13:05.9155894Z { 2024-08-20T22:13:05.9156445Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9157111Z "size": 139, 2024-08-20T22:13:05.9157733Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:05.9158455Z }, 2024-08-20T22:13:05.9158838Z { 2024-08-20T22:13:05.9159330Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9160012Z "size": 118, 2024-08-20T22:13:05.9160781Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:05.9161515Z }, 2024-08-20T22:13:05.9161822Z { 2024-08-20T22:13:05.9162448Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9163127Z "size": 4324672728, 2024-08-20T22:13:05.9163758Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:05.9164535Z }, 2024-08-20T22:13:05.9164900Z { 2024-08-20T22:13:05.9165430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9166104Z "size": 174, 2024-08-20T22:13:05.9166752Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:05.9167454Z }, 2024-08-20T22:13:05.9167867Z { 2024-08-20T22:13:05.9168364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9169030Z "size": 907, 2024-08-20T22:13:05.9169784Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:05.9170614Z }, 2024-08-20T22:13:05.9170923Z { 2024-08-20T22:13:05.9171530Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9172194Z "size": 701, 2024-08-20T22:13:05.9172814Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:05.9173570Z }, 2024-08-20T22:13:05.9173915Z { 2024-08-20T22:13:05.9174510Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9175125Z "size": 134, 2024-08-20T22:13:05.9175764Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:05.9176591Z }, 2024-08-20T22:13:05.9176931Z { 2024-08-20T22:13:05.9177432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9178157Z "size": 32, 2024-08-20T22:13:05.9178895Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:05.9179600Z }, 2024-08-20T22:13:05.9179996Z { 2024-08-20T22:13:05.9180535Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9181209Z "size": 158, 2024-08-20T22:13:05.9181857Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:05.9182587Z }, 2024-08-20T22:13:05.9182963Z { 2024-08-20T22:13:05.9183547Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9184188Z "size": 1841, 2024-08-20T22:13:05.9184819Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:05.9185584Z }, 2024-08-20T22:13:05.9185961Z { 2024-08-20T22:13:05.9186463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9187170Z "size": 7529776, 2024-08-20T22:13:05.9187843Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:05.9188533Z }, 2024-08-20T22:13:05.9188955Z { 2024-08-20T22:13:05.9189541Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9190190Z "size": 163, 2024-08-20T22:13:05.9190864Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:05.9191578Z }, 2024-08-20T22:13:05.9191931Z { 2024-08-20T22:13:05.9192539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9193157Z "size": 7943, 2024-08-20T22:13:05.9193805Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:05.9194589Z }, 2024-08-20T22:13:05.9194900Z { 2024-08-20T22:13:05.9195454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9196159Z "size": 8068, 2024-08-20T22:13:05.9196826Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:05.9197527Z }, 2024-08-20T22:13:05.9197922Z { 2024-08-20T22:13:05.9198460Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9199182Z "size": 302, 2024-08-20T22:13:05.9199778Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:05.9200762Z }, 2024-08-20T22:13:05.9201248Z { 2024-08-20T22:13:05.9201787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9202433Z "size": 7629638, 2024-08-20T22:13:05.9203159Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:05.9203945Z }, 2024-08-20T22:13:05.9204277Z { 2024-08-20T22:13:05.9204877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9205573Z "size": 108, 2024-08-20T22:13:05.9206280Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:05.9207064Z }, 2024-08-20T22:13:05.9207426Z { 2024-08-20T22:13:05.9208004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9208903Z "size": 54145661, 2024-08-20T22:13:05.9209561Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:05.9210277Z }, 2024-08-20T22:13:05.9210669Z { 2024-08-20T22:13:05.9211234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9211901Z "size": 473, 2024-08-20T22:13:05.9212614Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:05.9213339Z }, 2024-08-20T22:13:05.9213650Z { 2024-08-20T22:13:05.9214288Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9214957Z "size": 1374858766, 2024-08-20T22:13:05.9215625Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:05.9216365Z }, 2024-08-20T22:13:05.9216719Z { 2024-08-20T22:13:05.9217402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9218144Z "size": 106, 2024-08-20T22:13:05.9218760Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:05.9219518Z }, 2024-08-20T22:13:05.9219915Z { 2024-08-20T22:13:05.9220413Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9221099Z "size": 559, 2024-08-20T22:13:05.9221796Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:05.9222545Z }, 2024-08-20T22:13:05.9222858Z { 2024-08-20T22:13:05.9223449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9224131Z "size": 46248488, 2024-08-20T22:13:05.9224867Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:05.9225540Z }, 2024-08-20T22:13:05.9225913Z { 2024-08-20T22:13:05.9226497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9227131Z "size": 111, 2024-08-20T22:13:05.9227801Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:05.9228565Z }, 2024-08-20T22:13:05.9228934Z { 2024-08-20T22:13:05.9229432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9230128Z "size": 32, 2024-08-20T22:13:05.9230797Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:05.9231525Z }, 2024-08-20T22:13:05.9232023Z { 2024-08-20T22:13:05.9232601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9233254Z "size": 32, 2024-08-20T22:13:05.9233983Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:05.9234662Z }, 2024-08-20T22:13:05.9235004Z { 2024-08-20T22:13:05.9235630Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9236264Z "size": 32, 2024-08-20T22:13:05.9236915Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:05.9237706Z }, 2024-08-20T22:13:05.9238055Z { 2024-08-20T22:13:05.9238603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:05.9239332Z "size": 32, 2024-08-20T22:13:05.9239973Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:05.9240832Z } 2024-08-20T22:13:05.9241144Z ] 2024-08-20T22:13:05.9241492Z } 2024-08-20T22:13:05.9540673Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:05.9541180Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:05.9541771Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:05.9551038Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:05.9551547Z env: 2024-08-20T22:13:05.9551835Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:05.9553031Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.9554291Z ##[endgroup] 2024-08-20T22:13:05.9586381Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.9651460Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:05.9652046Z with: 2024-08-20T22:13:05.9653145Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.9654481Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.9655002Z env: 2024-08-20T22:13:05.9655284Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:05.9655638Z ##[endgroup] 2024-08-20T22:13:05.9672801Z ##[group]Run set -x 2024-08-20T22:13:05.9673300Z set -x 2024-08-20T22:13:05.9673603Z set +e 2024-08-20T22:13:05.9673902Z  2024-08-20T22:13:05.9674176Z login() { 2024-08-20T22:13:05.9674966Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:05.9675799Z } 2024-08-20T22:13:05.9676094Z  2024-08-20T22:13:05.9676409Z retry () { 2024-08-20T22:13:05.9676844Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:05.9677300Z } 2024-08-20T22:13:05.9677582Z  2024-08-20T22:13:05.9677910Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:05.9678333Z  2024-08-20T22:13:05.9678639Z set -e 2024-08-20T22:13:05.9679162Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:05.9679935Z # only pull docker image if it's not available locally 2024-08-20T22:13:05.9680818Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:05.9681558Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:05.9682006Z fi 2024-08-20T22:13:05.9690093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:05.9690609Z env: 2024-08-20T22:13:05.9690901Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:05.9692140Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:05.9693485Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.9694009Z ##[endgroup] 2024-08-20T22:13:05.9722820Z + set +e 2024-08-20T22:13:05.9723446Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.9724107Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:05.9726903Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:05.9729378Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:06.5073867Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:06.5074897Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:06.5075904Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:06.5076461Z 2024-08-20T22:13:06.5076590Z Login Succeeded 2024-08-20T22:13:06.5096218Z + set -e 2024-08-20T22:13:06.5097828Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:06.5257139Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:06.5259679Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:06.7746662Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:06.7747850Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:06.7748361Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:06.7749185Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:06.7749939Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:06.7750469Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:06.7750996Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:06.7751698Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:06.7752177Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:06.7752564Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:06.7752957Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:06.7753359Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:06.7754067Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:06.7754610Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:06.7755588Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:06.7756039Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:06.7756540Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:06.7757032Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:06.7757714Z 7ed32bc8e469: Waiting 2024-08-20T22:13:06.7758169Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:06.7758618Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:06.7759116Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:06.7759588Z ec1e7978c1fe: Waiting 2024-08-20T22:13:06.7760020Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:06.7760392Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:06.7761340Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:06.7761725Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:06.7762237Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:06.7762619Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:06.7763188Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:06.7763596Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:06.7763943Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:06.7764337Z a41a8d1c11c8: Waiting 2024-08-20T22:13:06.7764670Z ca426ffc9d2b: Waiting 2024-08-20T22:13:06.7764992Z d527cbbb87e3: Waiting 2024-08-20T22:13:06.7765376Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:06.7765745Z 0bf61114d37b: Waiting 2024-08-20T22:13:06.7766082Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:06.7766460Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:06.7766985Z b57676e46aee: Waiting 2024-08-20T22:13:06.7767333Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:06.7767740Z 1fef4bd19792: Waiting 2024-08-20T22:13:06.7768080Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:06.7768602Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:06.7769006Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:06.7769565Z 75b2abb16c68: Waiting 2024-08-20T22:13:06.7770020Z d8d1234baab3: Waiting 2024-08-20T22:13:06.7770368Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:06.7770777Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:06.7771348Z 019620471c25: Pulling fs layer 2024-08-20T22:13:06.7771741Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:06.7772473Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:06.7772850Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:06.7773219Z 33b639226eee: Waiting 2024-08-20T22:13:06.7773576Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:06.7774167Z 34c47d63c5de: Waiting 2024-08-20T22:13:06.7774522Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:06.7774944Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:06.7775313Z d3547bd43146: Waiting 2024-08-20T22:13:06.7775643Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:06.7776013Z 6658e558ebe4: Waiting 2024-08-20T22:13:06.7776350Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:06.7776716Z 829ed7d91e38: Waiting 2024-08-20T22:13:06.7777202Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:06.7777564Z 7c47122f4e09: Waiting 2024-08-20T22:13:06.7778022Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:06.7778540Z 2df849f72ba4: Waiting 2024-08-20T22:13:06.7778856Z 33444a5a7e01: Waiting 2024-08-20T22:13:06.7779195Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:06.7779560Z 560cd6122a38: Waiting 2024-08-20T22:13:06.7779889Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:06.7780484Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:06.7780868Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:06.7781369Z afc429e87b59: Waiting 2024-08-20T22:13:06.7781865Z f957c1624481: Pulling fs layer 2024-08-20T22:13:06.7782257Z 0479fbac3361: Waiting 2024-08-20T22:13:06.7782605Z 019620471c25: Waiting 2024-08-20T22:13:06.7783038Z f1d6874dee9c: Waiting 2024-08-20T22:13:06.7783354Z b119c4fc6c58: Waiting 2024-08-20T22:13:06.7783828Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:06.7784199Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:06.7784805Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:06.7785310Z 829c0e19f4b7: Waiting 2024-08-20T22:13:06.7785651Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:06.7786189Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:06.7786635Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:06.7787203Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:06.7787571Z 916c5e858016: Waiting 2024-08-20T22:13:06.7787879Z 5645520256fa: Waiting 2024-08-20T22:13:06.7788216Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:06.7788582Z 7754ee068b33: Waiting 2024-08-20T22:13:06.7788896Z f957c1624481: Waiting 2024-08-20T22:13:06.7789226Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:06.7789604Z c42b3602a1fb: Waiting 2024-08-20T22:13:06.7789946Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:06.7790315Z 5a4eb19d7219: Waiting 2024-08-20T22:13:06.7790812Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:06.7791180Z 4f4fb700ef54: Waiting 2024-08-20T22:13:06.7791516Z ee1273153675: Pulling fs layer 2024-08-20T22:13:06.7791875Z 78ee77eddef2: Waiting 2024-08-20T22:13:06.7792195Z 768c31dca016: Waiting 2024-08-20T22:13:06.7792516Z 11fad972ef6c: Waiting 2024-08-20T22:13:06.7792853Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:06.7793535Z 37513e9b43d7: Waiting 2024-08-20T22:13:06.7793877Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:06.7794384Z a8c1e85b5e14: Waiting 2024-08-20T22:13:06.7794724Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:06.7795102Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:06.7795471Z e56eeff2db93: Waiting 2024-08-20T22:13:06.7795953Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:06.7796312Z b10de02255c0: Waiting 2024-08-20T22:13:06.7796750Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:06.7797231Z eb09cbf3e416: Waiting 2024-08-20T22:13:06.7797573Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:06.7797953Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:06.7798343Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:06.7798710Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:06.7799027Z 0f085d4fa279: Waiting 2024-08-20T22:13:06.7799349Z 302c8c177944: Waiting 2024-08-20T22:13:06.7799664Z 385d0421d67c: Waiting 2024-08-20T22:13:06.7799982Z ee1273153675: Waiting 2024-08-20T22:13:06.7800295Z 1811ff4d3556: Waiting 2024-08-20T22:13:06.7801152Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:06.7801483Z 5d115b29b5d5: Waiting 2024-08-20T22:13:06.7801862Z d79fd7d52698: Waiting 2024-08-20T22:13:06.7802178Z 83808b2eff4d: Waiting 2024-08-20T22:13:06.7802632Z 9752d6fbbcab: Waiting 2024-08-20T22:13:06.7802958Z da6a838a66c7: Waiting 2024-08-20T22:13:06.7803270Z 02970347dff6: Waiting 2024-08-20T22:13:06.7803762Z cbb79a705353: Waiting 2024-08-20T22:13:06.7804182Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:06.7804516Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:06.7804836Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:06.9188855Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:06.9189458Z 224fe954d725: Download complete 2024-08-20T22:13:07.0080891Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:07.0081547Z d527cbbb87e3: Download complete 2024-08-20T22:13:07.1108835Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:07.1109502Z 7a2c55901189: Download complete 2024-08-20T22:13:07.1307907Z b57676e46aee: Verifying Checksum 2024-08-20T22:13:07.1308567Z b57676e46aee: Download complete 2024-08-20T22:13:07.2266601Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:13:07.2267117Z a41a8d1c11c8: Download complete 2024-08-20T22:13:07.3060580Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:07.3061192Z 0c1227890755: Download complete 2024-08-20T22:13:07.3912056Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:07.3912697Z 75722010b82e: Download complete 2024-08-20T22:13:07.4726545Z ec1e7978c1fe: Download complete 2024-08-20T22:13:07.5517578Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:08.3071730Z 7a2c55901189: Pull complete 2024-08-20T22:13:08.6329030Z 224fe954d725: Pull complete 2024-08-20T22:13:09.5321555Z 75722010b82e: Pull complete 2024-08-20T22:13:09.5524655Z d527cbbb87e3: Pull complete 2024-08-20T22:13:09.5707715Z b57676e46aee: Pull complete 2024-08-20T22:13:10.0674842Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:10.0675345Z ca426ffc9d2b: Download complete 2024-08-20T22:13:10.1612799Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:10.1613352Z 4373aa743caf: Download complete 2024-08-20T22:13:10.2404191Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:10.2405188Z 75b2abb16c68: Download complete 2024-08-20T22:13:10.3050446Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:13:10.3051028Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:11.4256637Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:11.4257123Z 0bf61114d37b: Download complete 2024-08-20T22:13:11.4865421Z 2df849f72ba4: Download complete 2024-08-20T22:13:11.5541333Z 34c47d63c5de: Download complete 2024-08-20T22:13:11.6466894Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:13:11.6467435Z 1fef4bd19792: Download complete 2024-08-20T22:13:20.4606595Z a8c1e85b5e14: Download complete 2024-08-20T22:13:20.5394463Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:20.5395063Z b119c4fc6c58: Download complete 2024-08-20T22:13:20.6141213Z 33b639226eee: Verifying Checksum 2024-08-20T22:13:20.6142179Z 33b639226eee: Download complete 2024-08-20T22:13:20.6870852Z d3547bd43146: Download complete 2024-08-20T22:13:20.7524329Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:20.7524968Z 5645520256fa: Download complete 2024-08-20T22:13:20.8178651Z 829ed7d91e38: Download complete 2024-08-20T22:13:22.0855438Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:22.0856127Z 385d0421d67c: Download complete 2024-08-20T22:13:22.1806257Z 7cbfce999d76: Download complete 2024-08-20T22:13:22.2598632Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:22.2599246Z 6658e558ebe4: Download complete 2024-08-20T22:13:22.3378205Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:13:22.3378733Z 33444a5a7e01: Download complete 2024-08-20T22:13:22.4257841Z 7c47122f4e09: Download complete 2024-08-20T22:13:22.5018419Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:22.5019041Z 1811ff4d3556: Download complete 2024-08-20T22:13:28.5396264Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:28.5396812Z 7754ee068b33: Download complete 2024-08-20T22:13:28.6207679Z 0479fbac3361: Verifying Checksum 2024-08-20T22:13:28.6208281Z 0479fbac3361: Download complete 2024-08-20T22:13:28.6903448Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:13:28.6904041Z f1d6874dee9c: Download complete 2024-08-20T22:13:29.1116522Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:29.1117061Z afc429e87b59: Download complete 2024-08-20T22:13:29.1831842Z 019620471c25: Download complete 2024-08-20T22:13:29.2866039Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:29.2867831Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:29.5502213Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:29.5502824Z 829c0e19f4b7: Download complete 2024-08-20T22:13:29.6267593Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:29.6268225Z 560cd6122a38: Download complete 2024-08-20T22:13:29.6994050Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:13:29.7694313Z da6a838a66c7: Download complete 2024-08-20T22:13:32.7242860Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:32.7243397Z 7ed32bc8e469: Download complete 2024-08-20T22:13:32.8004457Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:13:32.8004945Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:32.8775028Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:13:32.8775547Z 0f085d4fa279: Download complete 2024-08-20T22:13:33.4300930Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:33.4301392Z 4a0412373305: Download complete 2024-08-20T22:13:33.4994810Z 302c8c177944: Verifying Checksum 2024-08-20T22:13:33.4995249Z 302c8c177944: Download complete 2024-08-20T22:13:33.5638766Z 02970347dff6: Download complete 2024-08-20T22:13:33.6344749Z 916c5e858016: Download complete 2024-08-20T22:13:33.6773130Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:33.7401059Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:33.8443154Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:34.0996191Z 0c1227890755: Pull complete 2024-08-20T22:13:34.2672553Z d8d1234baab3: Pull complete 2024-08-20T22:13:40.3356491Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:40.3357003Z 6efb96fc87ea: Download complete 2024-08-20T22:13:40.4032672Z f957c1624481: Download complete 2024-08-20T22:13:40.4833514Z 768c31dca016: Verifying Checksum 2024-08-20T22:13:40.4834004Z 768c31dca016: Download complete 2024-08-20T22:13:40.5647179Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:13:40.5647797Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:40.5799259Z 4f4fb700ef54: Download complete 2024-08-20T22:13:40.6488319Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:13:40.6488807Z 78ee77eddef2: Download complete 2024-08-20T22:13:40.7419904Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:13:40.7420382Z 37513e9b43d7: Download complete 2024-08-20T22:13:40.8645240Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:40.8645689Z a7a5392e2bf6: Download complete 2024-08-20T22:13:40.9401187Z b10de02255c0: Download complete 2024-08-20T22:13:41.0123460Z e56eeff2db93: Verifying Checksum 2024-08-20T22:13:41.0124040Z e56eeff2db93: Download complete 2024-08-20T22:13:41.0894417Z 70ac3e38d0fd: Download complete 2024-08-20T22:13:41.1491128Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:13:41.1491609Z eb09cbf3e416: Download complete 2024-08-20T22:13:41.2825691Z ee1273153675: Verifying Checksum 2024-08-20T22:13:41.2826223Z ee1273153675: Download complete 2024-08-20T22:13:41.3494300Z e22979f9e0ee: Download complete 2024-08-20T22:13:41.9426434Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:41.9427053Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:42.0132282Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:13:42.0132783Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:07.7187993Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:07.7188491Z 5d115b29b5d5: Download complete 2024-08-20T22:14:07.8048037Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:07.8048498Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:07.8788113Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:07.8788645Z d79fd7d52698: Download complete 2024-08-20T22:14:09.1261027Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:09.1261642Z 9752d6fbbcab: Download complete 2024-08-20T22:14:09.1907700Z 83808b2eff4d: Download complete 2024-08-20T22:14:48.3271541Z cbb79a705353: Verifying Checksum 2024-08-20T22:14:48.9961067Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:14:48.9961558Z 11fad972ef6c: Download complete 2024-08-20T22:15:03.1243078Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:03.3571904Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:03.5845064Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:12.2776602Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:12.4944851Z 4373aa743caf: Pull complete 2024-08-20T22:15:12.7261073Z 75b2abb16c68: Pull complete 2024-08-20T22:15:12.9516363Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:15.4365972Z 0bf61114d37b: Pull complete 2024-08-20T22:15:15.5623873Z 2df849f72ba4: Pull complete 2024-08-20T22:15:15.7352609Z 34c47d63c5de: Pull complete 2024-08-20T22:15:15.8635992Z 1fef4bd19792: Pull complete 2024-08-20T22:16:14.9784626Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:15.0858256Z b119c4fc6c58: Pull complete 2024-08-20T22:16:15.2961698Z 33b639226eee: Pull complete 2024-08-20T22:16:15.5138710Z d3547bd43146: Pull complete 2024-08-20T22:16:15.6873999Z 5645520256fa: Pull complete 2024-08-20T22:16:15.9062748Z 829ed7d91e38: Pull complete 2024-08-20T22:16:19.5173511Z 385d0421d67c: Pull complete 2024-08-20T22:16:19.7463744Z 7cbfce999d76: Pull complete 2024-08-20T22:16:19.8505093Z 6658e558ebe4: Pull complete 2024-08-20T22:16:20.0804586Z 33444a5a7e01: Pull complete 2024-08-20T22:16:20.1811612Z 7c47122f4e09: Pull complete 2024-08-20T22:16:20.3489462Z 1811ff4d3556: Pull complete 2024-08-20T22:16:30.6590855Z 7754ee068b33: Pull complete 2024-08-20T22:16:30.8718521Z 0479fbac3361: Pull complete 2024-08-20T22:16:31.0871747Z f1d6874dee9c: Pull complete 2024-08-20T22:16:32.0467503Z afc429e87b59: Pull complete 2024-08-20T22:16:32.2253136Z 019620471c25: Pull complete 2024-08-20T22:16:32.3928682Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:32.8808715Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:33.0814940Z 560cd6122a38: Pull complete 2024-08-20T22:16:33.4576856Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:33.6484028Z da6a838a66c7: Pull complete 2024-08-20T22:17:12.5490398Z cbb79a705353: Pull complete 2024-08-20T22:17:12.7751226Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:12.9993363Z 0f085d4fa279: Pull complete 2024-08-20T22:17:15.3672372Z 4a0412373305: Pull complete 2024-08-20T22:17:15.3858348Z 302c8c177944: Pull complete 2024-08-20T22:17:15.4033013Z 02970347dff6: Pull complete 2024-08-20T22:17:15.4356596Z 916c5e858016: Pull complete 2024-08-20T22:17:15.4523865Z c42b3602a1fb: Pull complete 2024-08-20T22:18:37.4185062Z 11fad972ef6c: Pull complete 2024-08-20T22:18:37.6261634Z f957c1624481: Pull complete 2024-08-20T22:18:37.8465781Z 768c31dca016: Pull complete 2024-08-20T22:18:38.2625113Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:38.4886915Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:38.7114502Z 78ee77eddef2: Pull complete 2024-08-20T22:18:38.9124593Z 37513e9b43d7: Pull complete 2024-08-20T22:18:39.2843650Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:39.5090257Z b10de02255c0: Pull complete 2024-08-20T22:18:39.7317874Z e56eeff2db93: Pull complete 2024-08-20T22:18:39.9492503Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:40.1669749Z eb09cbf3e416: Pull complete 2024-08-20T22:18:41.5803931Z ee1273153675: Pull complete 2024-08-20T22:18:41.8153638Z e22979f9e0ee: Pull complete 2024-08-20T22:18:44.0297011Z 6c1f0aabce7c: Pull complete 2024-08-20T22:18:44.2591333Z 7ebe49c3a23e: Pull complete 2024-08-20T22:18:57.7441838Z 5d115b29b5d5: Pull complete 2024-08-20T22:18:57.9340614Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:18:58.1325491Z d79fd7d52698: Pull complete 2024-08-20T22:18:58.9766720Z 9752d6fbbcab: Pull complete 2024-08-20T22:18:59.1917238Z 83808b2eff4d: Pull complete 2024-08-20T22:19:00.1398993Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:00.1834534Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:00.2076335Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:00.2206556Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:00.2207504Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:00.2218773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:00.2219279Z env: 2024-08-20T22:19:00.2219571Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:00.2219926Z ##[endgroup] 2024-08-20T22:19:00.2442483Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:00.2443044Z with: 2024-08-20T22:19:00.2443333Z driver-version: 550.54.15 2024-08-20T22:19:00.2443672Z env: 2024-08-20T22:19:00.2443949Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:00.2444301Z ##[endgroup] 2024-08-20T22:19:00.2512887Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:00.2513449Z with: 2024-08-20T22:19:00.2513732Z timeout_minutes: 10 2024-08-20T22:19:00.2514070Z max_attempts: 3 2024-08-20T22:19:00.2547425Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:00.2581456Z retry_wait_seconds: 10 2024-08-20T22:19:00.2581836Z polling_interval_seconds: 1 2024-08-20T22:19:00.2582213Z warning_on_retry: true 2024-08-20T22:19:00.2582574Z continue_on_error: false 2024-08-20T22:19:00.2582916Z env: 2024-08-20T22:19:00.2583193Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:00.2583552Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:00.2583912Z ##[endgroup] 2024-08-20T22:19:00.3421007Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:00.3421679Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:00.3422237Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:00.7952245Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:00.7953299Z No packages marked for removal. 2024-08-20T22:19:00.8013857Z Dependencies resolved. 2024-08-20T22:19:00.8023576Z Nothing to do. 2024-08-20T22:19:00.8023889Z Complete! 2024-08-20T22:19:00.8877433Z + install_nvidia_driver_common 2024-08-20T22:19:00.8881708Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:00.8882181Z Before installing NVIDIA driver 2024-08-20T22:19:00.8883040Z + lspci 2024-08-20T22:19:00.9667001Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:00.9668547Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:00.9670733Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:00.9672369Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:00.9674043Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:00.9675351Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:00.9676102Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:00.9676933Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:00.9677530Z + lsmod 2024-08-20T22:19:00.9712699Z Module Size Used by 2024-08-20T22:19:00.9713132Z xt_conntrack 16384 1 2024-08-20T22:19:00.9713522Z nft_chain_nat 16384 3 2024-08-20T22:19:00.9713912Z xt_MASQUERADE 20480 1 2024-08-20T22:19:00.9714373Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:00.9714870Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:00.9715474Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:00.9716134Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:00.9716626Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:00.9717063Z xfrm_user 57344 1 2024-08-20T22:19:00.9717466Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:00.9717889Z xt_addrtype 16384 2 2024-08-20T22:19:00.9718278Z nft_compat 20480 4 2024-08-20T22:19:00.9718741Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:00.9719437Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:00.9720163Z br_netfilter 36864 0 2024-08-20T22:19:00.9720816Z bridge 307200 1 br_netfilter 2024-08-20T22:19:00.9721290Z stp 16384 1 bridge 2024-08-20T22:19:00.9721715Z llc 16384 2 bridge,stp 2024-08-20T22:19:00.9722142Z overlay 167936 0 2024-08-20T22:19:00.9722509Z tls 114688 0 2024-08-20T22:19:00.9722891Z nls_ascii 16384 1 2024-08-20T22:19:00.9723264Z nls_cp437 20480 1 2024-08-20T22:19:00.9723831Z vfat 24576 1 2024-08-20T22:19:00.9724201Z fat 86016 1 vfat 2024-08-20T22:19:00.9724595Z ena 167936 0 2024-08-20T22:19:00.9724971Z ptp 36864 1 ena 2024-08-20T22:19:00.9726099Z pps_core 24576 1 ptp 2024-08-20T22:19:00.9726507Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:00.9726996Z aesni_intel 393216 0 2024-08-20T22:19:00.9727384Z i8042 45056 0 2024-08-20T22:19:00.9727772Z serio 28672 3 i8042 2024-08-20T22:19:00.9728199Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:00.9728734Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:00.9729239Z button 24576 0 2024-08-20T22:19:00.9729614Z sunrpc 692224 1 2024-08-20T22:19:00.9729990Z sch_fq_codel 20480 17 2024-08-20T22:19:00.9730372Z dm_mod 188416 0 2024-08-20T22:19:00.9730739Z fuse 163840 1 2024-08-20T22:19:00.9731134Z dax 45056 1 dm_mod 2024-08-20T22:19:00.9731535Z configfs 57344 1 2024-08-20T22:19:00.9731911Z loop 36864 0 2024-08-20T22:19:00.9732283Z dmi_sysfs 20480 0 2024-08-20T22:19:00.9732682Z crc32_pclmul 16384 0 2024-08-20T22:19:00.9733080Z crc32c_intel 24576 0 2024-08-20T22:19:00.9733547Z efivarfs 24576 1 2024-08-20T22:19:00.9734034Z + modinfo nvidia 2024-08-20T22:19:00.9734964Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:00.9735752Z alias: char-major-195-* 2024-08-20T22:19:00.9736147Z version: 550.54.15 2024-08-20T22:19:00.9736503Z supported: external 2024-08-20T22:19:00.9736848Z license: NVIDIA 2024-08-20T22:19:00.9737238Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:00.9737736Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:00.9738209Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:00.9738687Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:00.9739201Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:00.9739704Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:00.9740209Z depends: i2c-core,drm 2024-08-20T22:19:00.9740570Z retpoline: Y 2024-08-20T22:19:00.9740883Z name: nvidia 2024-08-20T22:19:00.9741643Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:00.9742369Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:00.9743033Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:00.9743651Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:00.9744107Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:00.9744549Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:00.9745021Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:00.9745473Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:00.9745934Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:00.9746473Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:00.9747059Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:00.9747555Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:00.9747998Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:00.9748428Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:00.9748898Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:00.9749450Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:00.9750058Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:00.9750635Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:00.9751270Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:00.9751891Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:00.9752528Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:00.9753225Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:00.9753731Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:00.9754280Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:00.9754850Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:00.9755414Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:00.9755902Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:00.9756408Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:00.9756893Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:00.9757349Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:00.9757871Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:00.9758414Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:00.9758902Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:00.9759408Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:00.9759934Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:00.9760434Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:00.9761059Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:00.9761550Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:00.9761975Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:00.9762461Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:00.9762947Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:00.9763420Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:00.9763913Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:00.9764403Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:00.9764873Z parm: rm_firmware_active:charp 2024-08-20T22:19:00.9765301Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:00.9765694Z ++ command -v nvidia-smi 2024-08-20T22:19:00.9766105Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:00.9766476Z + set +e 2024-08-20T22:19:00.9767009Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:03.2241936Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:03.2242473Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:03.2243051Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:03.2243433Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:03.2243823Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:03.2244941Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:03.2245676Z + set -e 2024-08-20T22:19:03.2245993Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:03.2246582Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:03.2247285Z + post_install_nvidia_driver_common 2024-08-20T22:19:03.2248454Z + sudo modprobe nvidia 2024-08-20T22:19:03.3099928Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:03.3100904Z + lspci 2024-08-20T22:19:03.3101214Z After installing NVIDIA driver 2024-08-20T22:19:03.3217196Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:03.3218130Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:03.3219092Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:03.3219904Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:03.3220742Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:03.3221555Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:03.3222308Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:03.3223130Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:03.3223731Z + lsmod 2024-08-20T22:19:03.3251121Z Module Size Used by 2024-08-20T22:19:03.3251610Z nvidia_uvm 4706304 0 2024-08-20T22:19:03.3252130Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:19:03.3253016Z drm 602112 1 nvidia 2024-08-20T22:19:03.3253604Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:03.3254193Z backlight 24576 1 drm 2024-08-20T22:19:03.3254734Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:03.3255292Z xt_conntrack 16384 1 2024-08-20T22:19:03.3255795Z nft_chain_nat 16384 3 2024-08-20T22:19:03.3256312Z xt_MASQUERADE 20480 1 2024-08-20T22:19:03.3256925Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:03.3257614Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:03.3258398Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:03.3259101Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:03.3259568Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:03.3260009Z xfrm_user 57344 1 2024-08-20T22:19:03.3260406Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:03.3260835Z xt_addrtype 16384 2 2024-08-20T22:19:03.3261263Z nft_compat 20480 4 2024-08-20T22:19:03.3261867Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:03.3262563Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:03.3263109Z br_netfilter 36864 0 2024-08-20T22:19:03.3263516Z bridge 307200 1 br_netfilter 2024-08-20T22:19:03.3263968Z stp 16384 1 bridge 2024-08-20T22:19:03.3264388Z llc 16384 2 bridge,stp 2024-08-20T22:19:03.3264803Z overlay 167936 0 2024-08-20T22:19:03.3265169Z tls 114688 0 2024-08-20T22:19:03.3265537Z nls_ascii 16384 1 2024-08-20T22:19:03.3265917Z nls_cp437 20480 1 2024-08-20T22:19:03.3266289Z vfat 24576 1 2024-08-20T22:19:03.3266654Z fat 86016 1 vfat 2024-08-20T22:19:03.3267045Z ena 167936 0 2024-08-20T22:19:03.3267423Z ptp 36864 1 ena 2024-08-20T22:19:03.3267820Z pps_core 24576 1 ptp 2024-08-20T22:19:03.3268215Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:03.3268601Z aesni_intel 393216 0 2024-08-20T22:19:03.3268977Z i8042 45056 0 2024-08-20T22:19:03.3269468Z serio 28672 3 i8042 2024-08-20T22:19:03.3270049Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:03.3270985Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:03.3271512Z button 24576 0 2024-08-20T22:19:03.3271876Z sunrpc 692224 1 2024-08-20T22:19:03.3272343Z sch_fq_codel 20480 17 2024-08-20T22:19:03.3272834Z dm_mod 188416 0 2024-08-20T22:19:03.3273312Z fuse 163840 1 2024-08-20T22:19:03.3273790Z dax 45056 1 dm_mod 2024-08-20T22:19:03.3274320Z configfs 57344 1 2024-08-20T22:19:03.3274757Z loop 36864 0 2024-08-20T22:19:03.3275128Z dmi_sysfs 20480 0 2024-08-20T22:19:03.3275505Z crc32_pclmul 16384 0 2024-08-20T22:19:03.3275882Z crc32c_intel 24576 0 2024-08-20T22:19:03.3276319Z efivarfs 24576 1 2024-08-20T22:19:03.3276682Z + modinfo nvidia 2024-08-20T22:19:03.3277387Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:03.3278114Z alias: char-major-195-* 2024-08-20T22:19:03.3278505Z version: 550.54.15 2024-08-20T22:19:03.3278849Z supported: external 2024-08-20T22:19:03.3279200Z license: NVIDIA 2024-08-20T22:19:03.3279585Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:03.3280075Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:03.3280531Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:03.3281136Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:03.3281644Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:03.3282284Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:03.3282788Z depends: i2c-core,drm 2024-08-20T22:19:03.3283155Z retpoline: Y 2024-08-20T22:19:03.3283467Z name: nvidia 2024-08-20T22:19:03.3284088Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:03.3284791Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:03.3285457Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:03.3286060Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:03.3286513Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:03.3286949Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:03.3287419Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:03.3287865Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:03.3288306Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:03.3288847Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:03.3289426Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:03.3289916Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:03.3290347Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:03.3290770Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:03.3291240Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:03.3291782Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:03.3292375Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:03.3292951Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:03.3293576Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:03.3294184Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:03.3294805Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:03.3295416Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:03.3295913Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:03.3296489Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:03.3297075Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:03.3297582Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:03.3298056Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:03.3298550Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:03.3299026Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:03.3301268Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:03.3301826Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:03.3302366Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:03.3302863Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:03.3303367Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:03.3303883Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:03.3304376Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:03.3304898Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:03.3305397Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:03.3305818Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:03.3306294Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:03.3306827Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:03.3307296Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:03.3307793Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:03.3308274Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:03.3308740Z parm: rm_firmware_active:charp 2024-08-20T22:19:03.3309150Z + set +e 2024-08-20T22:19:03.3309481Z + nvidia-smi 2024-08-20T22:19:04.8814382Z Tue Aug 20 22:19:04 2024 2024-08-20T22:19:04.8815324Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:04.8816186Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:04.8816985Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:04.8818165Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:04.8819096Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:04.8819809Z | | | MIG M. | 2024-08-20T22:19:04.8820372Z |=========================================+========================+======================| 2024-08-20T22:19:04.8915243Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:04.8916000Z | 0% 29C P0 56W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:04.8916646Z | | | N/A | 2024-08-20T22:19:04.8917370Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:04.8918081Z 2024-08-20T22:19:04.8918764Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:04.8919405Z | Processes: | 2024-08-20T22:19:04.8920128Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:04.8920925Z | ID ID Usage | 2024-08-20T22:19:04.8921507Z |=========================================================================================| 2024-08-20T22:19:04.8922173Z | No running processes found | 2024-08-20T22:19:04.8922948Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:05.4993227Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:07.0517649Z NVIDIA A10G 2024-08-20T22:19:07.4930422Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:07.4931103Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:07.4932016Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:07.4932493Z + set -e 2024-08-20T22:19:07.4932801Z INFO: Ignoring allowed status 0 2024-08-20T22:19:07.4939208Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:07.4943399Z + sudo yum install -y yum-utils 2024-08-20T22:19:07.9354796Z Last metadata expiration check: 0:09:36 ago on Tue Aug 20 22:09:31 2024. 2024-08-20T22:19:07.9578752Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:07.9881300Z Dependencies resolved. 2024-08-20T22:19:08.0007311Z Nothing to do. 2024-08-20T22:19:08.0007629Z Complete! 2024-08-20T22:19:08.0841411Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:08.0842624Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:08.0843999Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:08.3898284Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:08.4664541Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:08.9475864Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:19:08.9703293Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:08.9999719Z Dependencies resolved. 2024-08-20T22:19:09.0127960Z Nothing to do. 2024-08-20T22:19:09.0128278Z Complete! 2024-08-20T22:19:09.1057459Z + sudo systemctl restart docker 2024-08-20T22:20:04.8890894Z Tue Aug 20 22:20:04 2024 2024-08-20T22:20:04.8891803Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:04.8892686Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:04.8893926Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:04.8894747Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:04.8895669Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:04.8896402Z | | | MIG M. | 2024-08-20T22:20:04.8897053Z |=========================================+========================+======================| 2024-08-20T22:20:04.9007979Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:04.9008714Z | 0% 30C P0 56W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:20:04.9009364Z | | | N/A | 2024-08-20T22:20:04.9010173Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:04.9010777Z 2024-08-20T22:20:04.9011447Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:04.9012084Z | Processes: | 2024-08-20T22:20:04.9012798Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:04.9013503Z | ID ID Usage | 2024-08-20T22:20:04.9014077Z |=========================================================================================| 2024-08-20T22:20:04.9014728Z | No running processes found | 2024-08-20T22:20:04.9015494Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:05.3826285Z Command completed after 1 attempt(s). 2024-08-20T22:20:05.3909509Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:05.3910313Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:05.3911031Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:05.3911705Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:05.3926507Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:05.3927013Z env: 2024-08-20T22:20:05.3927294Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:05.3927760Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:05.3928246Z ##[endgroup] 2024-08-20T22:20:05.6764081Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:06.0982186Z Collecting psutil==5.9.1 2024-08-20T22:20:06.1351705Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:06.1874934Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:06.1921182Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:06.2797420Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:06.4386606Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:06.6403187Z Prepare all required actions 2024-08-20T22:20:06.6403702Z Getting action download info 2024-08-20T22:20:06.7421583Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:06.9532849Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:07.1016063Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:07.1016564Z with: 2024-08-20T22:20:07.1016912Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:07.1017548Z s3-bucket: gha-artifacts 2024-08-20T22:20:07.1017903Z env: 2024-08-20T22:20:07.1018193Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:07.1018663Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:07.1019161Z ##[endgroup] 2024-08-20T22:20:07.1053911Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:07.1054377Z with: 2024-08-20T22:20:07.1054749Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:07.1055210Z s3-bucket: gha-artifacts 2024-08-20T22:20:07.1055571Z region: us-east-1 2024-08-20T22:20:07.1055882Z env: 2024-08-20T22:20:07.1056162Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:07.1056633Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:07.1057145Z ##[endgroup] 2024-08-20T22:20:07.5766951Z (node:52551) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:07.5767659Z 2024-08-20T22:20:07.5767937Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:07.5768683Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:07.5769581Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:07.6499713Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:07.6501079Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0752473Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0758770Z Artifact download has finished successfully 2024-08-20T22:20:25.1117830Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:25.1118278Z unzip -o artifacts.zip 2024-08-20T22:20:25.1127418Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:25.1127933Z env: 2024-08-20T22:20:25.1128218Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:25.1128676Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:25.1129192Z ##[endgroup] 2024-08-20T22:20:25.1209783Z Archive: artifacts.zip 2024-08-20T22:20:25.1210724Z creating: dist/ 2024-08-20T22:20:27.2499870Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:27.2500881Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:27.2501505Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:27.2502277Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:27.2503145Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.2509578Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.2510546Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.2511529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.2512570Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.2513580Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.2514719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.2516565Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.2517625Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.2518647Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.2520115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.2521996Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.2524351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.2525826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.2527145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.2528471Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.2529551Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.2530593Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.2570044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.2610117Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.2611611Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.2656998Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.2658539Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.2660035Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.2661562Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.2663066Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.2664563Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.2666014Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.2667452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.2668850Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.2670174Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.2671449Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.2672709Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.2673983Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.2675270Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.2676521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.2734586Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.2794683Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.2795953Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.2796988Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.2797890Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.2798824Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.2799981Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:27.2801541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2802751Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2803894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:27.2804963Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:27.2806090Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2807203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:27.2808317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2809441Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:27.2810536Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:27.2826716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:27.2970634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:27.2971698Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:27.2972831Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2974083Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2975344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:27.2976479Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:27.2977653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2978838Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:27.2980012Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2981193Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:27.2982357Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:27.2999823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:27.3083871Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:27.3085822Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3088042Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3090407Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3092310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3094157Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3095208Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:27.3096100Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:27.3096916Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:27.3097886Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:27.3215004Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:27.3278903Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:27.3280472Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:27.3282065Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:27.3283745Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3286965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3287953Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3288924Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3289938Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3290938Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3292197Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3293519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3294559Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3295595Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3297015Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3299593Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3301342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3303030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3304495Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3305934Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3307093Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3308127Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3347561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3387918Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3389429Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3435648Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3437418Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3439427Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3441415Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3443055Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3444823Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3446695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3448294Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3449874Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3451204Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3452478Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3453723Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3454997Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3456238Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3457499Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3510738Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.3571409Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.3572742Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.3574034Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.3575109Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.3576244Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.3577439Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:27.3578745Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:27.3580184Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:27.3581561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:27.3582880Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:27.3584030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:27.3585195Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:27.3586370Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:27.3587537Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:27.3588692Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:27.3604433Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:27.3668928Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:27.3670434Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3671736Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3672867Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3673969Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3675075Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3676005Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:27.3677025Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:27.3678043Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:27.3678994Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:27.3729404Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:27.3730313Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:27.3731114Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:27.3732030Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3738516Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3739727Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3740887Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3741996Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3743174Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3744536Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3746456Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3748039Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3749540Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3751325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3753109Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3754987Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3756445Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3758040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3759670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3760892Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3762185Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3802786Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3842354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3844123Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3889408Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3891100Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3892855Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3894904Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3896685Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3898381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3900202Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3902181Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3903869Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3905407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3906853Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3908174Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3909505Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3910797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3912107Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3966856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.4027588Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.4029039Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.4030317Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.4031453Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.4032709Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.4033957Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:27.4035331Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.4036843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.4038325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:27.4039716Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:27.4041055Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.4042459Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:27.4043776Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.4045146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:27.4046351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:27.4047652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:27.4171731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:27.4173180Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:27.4174713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.4176292Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.4177797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:27.4179277Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:27.4180754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.4182057Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:27.4183362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.4184691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:27.4185988Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:27.4203152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:27.4258765Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:27.4260394Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.4261751Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.4262938Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.4264150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.4265272Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.4266270Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:27.4267589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:27.4268746Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:27.4269902Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:27.4373649Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:27.4417191Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:27.4417851Z creating: build/lib/ 2024-08-20T22:20:27.4506822Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:27.4964119Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:27.4973735Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:27.4982036Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:27.4990305Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:27.4990985Z inflating: build/lib/libclog.a 2024-08-20T22:20:27.5009857Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:27.5012680Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:27.5077820Z inflating: build/lib/libgtest.a 2024-08-20T22:20:27.5153521Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:27.5218305Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:27.5226194Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:27.5254123Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:27.5382173Z inflating: build/lib/libgloo.a 2024-08-20T22:20:27.5403569Z inflating: build/lib/libfmt.a 2024-08-20T22:20:27.5498657Z inflating: build/lib/libc10.so 2024-08-20T22:20:27.5500614Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:27.5502371Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:27.5521953Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:27.6027671Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:27.6045892Z inflating: build/lib/libgmock.a 2024-08-20T22:20:27.6046613Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:27.6047505Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:27.6437476Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:28.6635608Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:28.7211310Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:28.7269415Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:28.7270538Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:28.8531273Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:28.8788137Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:28.9296260Z inflating: build/lib/libkineto.a 2024-08-20T22:20:28.9481346Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:28.9524498Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:29.0219028Z inflating: build/lib/libonnx.a 2024-08-20T22:20:31.4758967Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:31.4764001Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:31.4767674Z inflating: build/lib/libshm.so 2024-08-20T22:20:33.5490348Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:33.5491656Z inflating: build/lib/libtorch.so 2024-08-20T22:20:33.5494809Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:34.3718757Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:34.5720268Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:34.5792679Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:34.5814145Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:34.5840603Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:34.5866178Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:34.5902067Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:34.5903322Z creating: build/bin/ 2024-08-20T22:20:34.5952379Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:34.6004429Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:34.6055955Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:34.6115756Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:34.6169624Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:34.6219273Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:34.6271054Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:34.6326297Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:34.6382716Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:34.6439371Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:34.6511132Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:34.6564727Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:34.6614396Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:34.6664779Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:34.6716159Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:34.6772999Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:34.6828705Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:34.6879250Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:34.6935487Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:34.6987448Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:34.7039426Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:34.7088572Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:34.7140910Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:34.7196692Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:34.7248522Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:34.7305190Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:34.7361218Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:34.7414572Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:34.7465940Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:34.7516202Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:34.7681944Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:34.7732989Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:34.7787061Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:34.7844995Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:34.7908187Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:34.7983536Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:34.8135707Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:34.8190236Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:34.8242660Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:34.8295242Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:34.8346307Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:34.8405903Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:34.8454626Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:34.8511702Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:34.8962401Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:34.9412282Z inflating: build/bin/protoc 2024-08-20T22:20:34.9466431Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:34.9519365Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:34.9572781Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:34.9626567Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:34.9679833Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:34.9733444Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:34.9782926Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:34.9836609Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:35.0169075Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:35.0516346Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:35.0878115Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:35.0932059Z inflating: build/bin/BackoffTest 2024-08-20T22:20:35.0985409Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:35.1042566Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:35.1095787Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:35.1109846Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:35.1164455Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:35.1168753Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:35.1171812Z inflating: build/bin/example_allreduce 2024-08-20T22:20:35.1228067Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:35.1297393Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:35.1300186Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:35.1368046Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:35.1377284Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:35.1725935Z inflating: build/bin/test_lazy 2024-08-20T22:20:35.2915912Z inflating: build/bin/test_api 2024-08-20T22:20:35.2990517Z inflating: build/bin/Dict_test 2024-08-20T22:20:35.3043508Z inflating: build/bin/Dimname_test 2024-08-20T22:20:35.3109245Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:35.3167124Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:35.3227100Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:35.3286608Z inflating: build/bin/atest 2024-08-20T22:20:35.3350600Z inflating: build/bin/basic 2024-08-20T22:20:35.3406238Z inflating: build/bin/broadcast_test 2024-08-20T22:20:35.3457331Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:35.3516149Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:35.3569957Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:35.3664614Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:35.3714800Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:35.3766316Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:35.3824741Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:35.3880778Z inflating: build/bin/half_test 2024-08-20T22:20:35.3980464Z inflating: build/bin/ivalue_test 2024-08-20T22:20:35.4030598Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:35.4085596Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:35.4140485Z inflating: build/bin/memory_format_test 2024-08-20T22:20:35.4193840Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:35.4247771Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:35.4304299Z inflating: build/bin/native_test 2024-08-20T22:20:35.4355494Z inflating: build/bin/operator_name_test 2024-08-20T22:20:35.4407823Z inflating: build/bin/operators_test 2024-08-20T22:20:35.4460278Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:35.4528102Z inflating: build/bin/pow_test 2024-08-20T22:20:35.4586485Z inflating: build/bin/quantized_test 2024-08-20T22:20:35.4637539Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:35.4689055Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:35.4747042Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:35.4805844Z inflating: build/bin/scalar_test 2024-08-20T22:20:35.4858167Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:35.4911621Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:35.4992014Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:35.5046076Z inflating: build/bin/test_parallel 2024-08-20T22:20:35.5049228Z inflating: build/bin/thread_init_test 2024-08-20T22:20:35.5105743Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:35.5166493Z inflating: build/bin/type_test 2024-08-20T22:20:35.5219496Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:35.5220803Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:35.5291210Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:35.5343472Z inflating: build/bin/weakref_test 2024-08-20T22:20:35.5395418Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:35.5448332Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:35.5508604Z inflating: build/bin/IListRef_test 2024-08-20T22:20:35.5616052Z inflating: build/bin/List_test 2024-08-20T22:20:35.5683399Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:35.5807151Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:35.5906493Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:35.6036566Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:35.6140399Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:35.6202812Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:35.6299431Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:35.6351207Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:35.6408137Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:35.6457578Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:35.6762614Z inflating: build/bin/op_registration_test 2024-08-20T22:20:35.6827863Z inflating: build/bin/inline_container_test 2024-08-20T22:20:35.6880651Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:35.6934551Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:35.6994503Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:35.7050028Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:35.7121132Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:35.7180713Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:35.7230915Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:35.7289056Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:35.7340405Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:35.7405923Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:35.7463173Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:35.7513580Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:35.7565896Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:35.7615284Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:35.7667394Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:35.7720809Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:35.7771313Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:35.7832752Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:35.7882766Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:35.7935476Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:35.7950062Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:35.8017061Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:35.8075027Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:35.8139601Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:35.8202518Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:35.9036625Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:35.9623654Z inflating: build/bin/test_jit 2024-08-20T22:20:35.9624269Z creating: .additional_ci_files/ 2024-08-20T22:20:35.9685136Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:35.9926203Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:35.9962785Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:35.9963190Z rm artifacts.zip 2024-08-20T22:20:35.9972254Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:35.9972767Z env: 2024-08-20T22:20:35.9973046Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:35.9973511Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:35.9974012Z ##[endgroup] 2024-08-20T22:20:36.1205705Z ##[group]Run df -H 2024-08-20T22:20:36.1206040Z df -H 2024-08-20T22:20:36.1214918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.1215435Z env: 2024-08-20T22:20:36.1215719Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.1216191Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.1216707Z ##[endgroup] 2024-08-20T22:20:36.1267049Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:36.1267749Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:36.1268260Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:36.1268753Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:36.1269469Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:36.1270252Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:36.1270760Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:36.1273616Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:36.1314433Z Prepare all required actions 2024-08-20T22:20:36.1314880Z Getting action download info 2024-08-20T22:20:36.2373078Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:36.2373562Z with: 2024-08-20T22:20:36.2373825Z env: 2024-08-20T22:20:36.2374110Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2374579Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2375242Z ##[endgroup] 2024-08-20T22:20:36.2409725Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:36.2410182Z with: 2024-08-20T22:20:36.2410459Z name: td_results 2024-08-20T22:20:36.2410776Z s3-bucket: gha-artifacts 2024-08-20T22:20:36.2411135Z region: us-east-1 2024-08-20T22:20:36.2411443Z env: 2024-08-20T22:20:36.2411729Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.2412186Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.2412684Z ##[endgroup] 2024-08-20T22:20:36.7261967Z (node:52575) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:36.7263503Z 2024-08-20T22:20:36.7264046Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:36.7265512Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:36.7267243Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:36.7895503Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:36.7902962Z Artifact download has finished successfully 2024-08-20T22:20:36.8231626Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:36.8232128Z mkdir -p .additional_ci_files 2024-08-20T22:20:36.8232710Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:36.8241808Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.8242328Z env: 2024-08-20T22:20:36.8242617Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8243112Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8243621Z ##[endgroup] 2024-08-20T22:20:36.8299826Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:36.8312967Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:36.8348569Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:36.8349073Z .github/scripts/parse_ref.py 2024-08-20T22:20:36.8358027Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:36.8358390Z env: 2024-08-20T22:20:36.8358671Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8359149Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8359654Z ##[endgroup] 2024-08-20T22:20:36.8646951Z Prepare all required actions 2024-08-20T22:20:36.8686324Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:36.8686800Z with: 2024-08-20T22:20:36.8687385Z github-token: *** 2024-08-20T22:20:36.8687709Z env: 2024-08-20T22:20:36.8687999Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8688475Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8688981Z ##[endgroup] 2024-08-20T22:20:36.8706806Z ##[group]Run set -eux 2024-08-20T22:20:36.8707151Z set -eux 2024-08-20T22:20:36.8707751Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:36.8716349Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.8716874Z env: 2024-08-20T22:20:36.8717162Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.8717634Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.8718290Z GITHUB_TOKEN: *** 2024-08-20T22:20:36.8718617Z ##[endgroup] 2024-08-20T22:20:36.8748158Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-0284de2732bb465d7 2024-08-20T22:20:38.5818019Z setting job-id=29026528773 2024-08-20T22:20:38.5818994Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.6040963Z Prepare all required actions 2024-08-20T22:20:38.6041488Z Getting action download info 2024-08-20T22:20:38.7179629Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:38.7180105Z with: 2024-08-20T22:20:38.7180569Z github-token: *** 2024-08-20T22:20:38.7190103Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:38.7200233Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.7201354Z env: 2024-08-20T22:20:38.7201646Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.7202107Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.7202604Z ##[endgroup] 2024-08-20T22:20:38.7272405Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:38.7272964Z with: 2024-08-20T22:20:38.7273239Z shell: bash 2024-08-20T22:20:38.7273552Z timeout_minutes: 10 2024-08-20T22:20:38.7273882Z max_attempts: 5 2024-08-20T22:20:38.7274207Z retry_wait_seconds: 30 2024-08-20T22:20:38.7275341Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.7276536Z polling_interval_seconds: 1 2024-08-20T22:20:38.7276925Z warning_on_retry: true 2024-08-20T22:20:38.7277278Z continue_on_error: false 2024-08-20T22:20:38.7277620Z env: 2024-08-20T22:20:38.7277901Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.7278353Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.7279013Z GITHUB_TOKEN: *** 2024-08-20T22:20:38.7279341Z ##[endgroup] 2024-08-20T22:20:38.8024629Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:39.0414650Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:39.1897872Z Collecting requests==2.27.1 2024-08-20T22:20:39.2264822Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:39.4230938Z Collecting pyyaml==6.0.1 2024-08-20T22:20:39.4301580Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:39.5188468Z Collecting certifi>=2017.4.17 2024-08-20T22:20:39.5239027Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:39.5306760Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:39.5314041Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:39.8315524Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:39.8365545Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:39.9186228Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:40.0373054Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:40.7818220Z Command completed after 1 attempt(s). 2024-08-20T22:20:40.7876027Z ##[group]Run set -x 2024-08-20T22:20:40.7876372Z set -x 2024-08-20T22:20:40.7876674Z  2024-08-20T22:20:40.7877238Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.7877943Z # in runner workspace 2024-08-20T22:20:40.7878482Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:40.7887657Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.7888174Z env: 2024-08-20T22:20:40.7888453Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.7888923Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.7889418Z ##[endgroup] 2024-08-20T22:20:40.7920237Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:40.8158923Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.8159485Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.8159966Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:40.8160371Z  2024-08-20T22:20:40.8160934Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.8161762Z # in runner workspace 2024-08-20T22:20:40.8162370Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:40.8163024Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:40.8163498Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:40.8173125Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:40.8182858Z  --selected-test-configs "" \ 2024-08-20T22:20:40.8183335Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:40.8183769Z  --tag "${TAG}" \ 2024-08-20T22:20:40.8184173Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:40.8184626Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:40.8185068Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:40.8193839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.8194356Z env: 2024-08-20T22:20:40.8194646Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.8195112Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.8195814Z GITHUB_TOKEN: *** 2024-08-20T22:20:40.8196559Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.8197365Z PR_NUMBER: 2024-08-20T22:20:40.8197676Z TAG: ciflow/inductor/133712 2024-08-20T22:20:40.8198052Z EVENT_NAME: push 2024-08-20T22:20:40.8198365Z SCHEDULE: 2024-08-20T22:20:40.8198656Z HEAD_BRANCH: 2024-08-20T22:20:40.8198968Z ##[endgroup] 2024-08-20T22:20:40.8228960Z Workflow: inductor 2024-08-20T22:20:40.8230019Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.0571954Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:41.2744221Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:41.2744706Z echo "Filtered matrix:" 2024-08-20T22:20:41.2754085Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:41.2763913Z  2024-08-20T22:20:41.2764196Z echo 2024-08-20T22:20:41.2764591Z echo "Is the current job unstable? False" 2024-08-20T22:20:41.2765050Z  2024-08-20T22:20:41.2765328Z echo 2024-08-20T22:20:41.2765700Z echo "Is keep-going label set? False" 2024-08-20T22:20:41.2766152Z  2024-08-20T22:20:41.2766427Z echo 2024-08-20T22:20:41.2766752Z echo "Renabled issues? " 2024-08-20T22:20:41.2775742Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.2776244Z env: 2024-08-20T22:20:41.2776533Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.2777005Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.2777502Z ##[endgroup] 2024-08-20T22:20:41.2807381Z Filtered matrix: 2024-08-20T22:20:41.2818674Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:41.2827972Z 2024-08-20T22:20:41.2828133Z Is the current job unstable? False 2024-08-20T22:20:41.2828429Z 2024-08-20T22:20:41.2828797Z Is keep-going label set? False 2024-08-20T22:20:41.2829062Z 2024-08-20T22:20:41.2829194Z Renabled issues? 2024-08-20T22:20:41.2872080Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.2872810Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.2881057Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.2881641Z env: 2024-08-20T22:20:41.2881923Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.2882389Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.2882941Z JOB_TIMEOUT: 240 2024-08-20T22:20:41.2883249Z ##[endgroup] 2024-08-20T22:20:41.2965519Z ##[group]Run set -x 2024-08-20T22:20:41.2965921Z set -x 2024-08-20T22:20:41.2966224Z  2024-08-20T22:20:41.2966588Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:41.2967135Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:41.2967876Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:41.2968400Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:41.2968820Z else 2024-08-20T22:20:41.2969168Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.2969600Z fi 2024-08-20T22:20:41.2969883Z  2024-08-20T22:20:41.2970359Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:41.2971126Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:41.2971827Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:41.2972431Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:41.2972904Z container_name=$(docker run \ 2024-08-20T22:20:41.2973334Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:41.2973725Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:41.2974136Z  -e PR_NUMBER \ 2024-08-20T22:20:41.2974509Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:41.2974915Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:41.2975333Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:41.2975731Z  -e GITHUB_JOB \ 2024-08-20T22:20:41.2976095Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:41.2976494Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:41.2976912Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:41.2977319Z  -e JOB_ID \ 2024-08-20T22:20:41.2977661Z  -e JOB_NAME \ 2024-08-20T22:20:41.2978017Z  -e BASE_SHA \ 2024-08-20T22:20:41.2978367Z  -e BRANCH \ 2024-08-20T22:20:41.2978701Z  -e SHA1 \ 2024-08-20T22:20:41.2979053Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:41.2979467Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:41.2979856Z  -e SHARD_NUMBER \ 2024-08-20T22:20:41.2980228Z  -e TEST_CONFIG \ 2024-08-20T22:20:41.2980607Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:41.2981017Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:41.2981443Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:41.2981874Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:41.2982286Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:41.2982692Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:41.2983072Z  -e NO_TD \ 2024-08-20T22:20:41.2983417Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:41.2983807Z  -e PR_LABELS \ 2024-08-20T22:20:41.2984226Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:41.2984686Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:41.2985096Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:41.2985518Z  -e XLA_CUDA \ 2024-08-20T22:20:41.2985931Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:41.2986450Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:41.2986987Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:41.2987525Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:41.2988010Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:41.2988469Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:41.2988924Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:41.2989418Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:41.2989987Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:41.2990459Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:41.2990862Z  --ipc=host \ 2024-08-20T22:20:41.2991228Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:41.2991634Z  --tty \ 2024-08-20T22:20:41.2991950Z  --detach \ 2024-08-20T22:20:41.2992443Z  --name="${container_name}" \ 2024-08-20T22:20:41.2992878Z  --user jenkins \ 2024-08-20T22:20:41.2993381Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:41.2993951Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:41.2994486Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:41.2994842Z ) 2024-08-20T22:20:41.2995264Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:41.2996203Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:41.2997193Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:41.2998127Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:41.3007001Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:41.3007355Z env: 2024-08-20T22:20:41.3007635Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.3008107Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.3008726Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:41.3009237Z PR_NUMBER: 2024-08-20T22:20:41.3009568Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:41.3010004Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:41.3010367Z GITHUB_JOB: test 2024-08-20T22:20:41.3010686Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:41.3011060Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:41.3011419Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:41.3011760Z JOB_ID: 29026528773 2024-08-20T22:20:41.3012504Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.3013321Z BRANCH: 2024-08-20T22:20:41.3013665Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.3014203Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.3014719Z TEST_CONFIG: dynamic_inductor_huggingface 2024-08-20T22:20:41.3015171Z SHARD_NUMBER: 1 2024-08-20T22:20:41.3015491Z NUM_TEST_SHARDS: 1 2024-08-20T22:20:41.3015824Z REENABLED_ISSUES: 2024-08-20T22:20:41.3016174Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:41.3016572Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:41.3016939Z TEST_SHOWLOCALS: False 2024-08-20T22:20:41.3017287Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:41.3017628Z NO_TD: False 2024-08-20T22:20:41.3017936Z TD_DISTRIBUTED: False 2024-08-20T22:20:41.3018361Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:41.3018861Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:41.3019256Z SHM_SIZE: 2g 2024-08-20T22:20:41.3020382Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:41.3021610Z XLA_CUDA: 2024-08-20T22:20:41.3022096Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:41.3022743Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:41.3023191Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:41.3023610Z DASHBOARD_TAG: 2024-08-20T22:20:41.3024141Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:41.3024529Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:41.3024918Z ##[endgroup] 2024-08-20T22:20:41.3054045Z + [[ dynamic_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:41.3054858Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:41.3055390Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.3062975Z +++ nproc --ignore=2 2024-08-20T22:20:41.3094776Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:52.6013199Z + container_name=b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T22:20:52.6018034Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:52.6021148Z + docker exec -i b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:52.7774273Z + echo DOCKER_CONTAINER_ID=b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T22:20:52.7779148Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:52.7782319Z + docker exec -t b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:53.1742011Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:53.4986290Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:53.4988812Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:53.4991652Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:53.4994014Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:53.4998368Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:53.5002653Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:53.5019450Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:53.5032289Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:53.5049824Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:53.5433418Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:20:54.1180153Z Installing collected packages: torch 2024-08-20T22:21:03.9479511Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:03.9481686Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:03.9482443Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:04.0293024Z + export TERM=vt100 2024-08-20T22:21:04.0293422Z + TERM=vt100 2024-08-20T22:21:04.0295860Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:04.0307593Z + source .ci/pytorch/common.sh 2024-08-20T22:21:04.0311620Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:04.0320965Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:04.0322470Z +++ declare -f -t trap_add 2024-08-20T22:21:04.0329206Z ++ set -ex 2024-08-20T22:21:04.0329873Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:04.0330467Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:04.0330995Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:04.0333670Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:04.0352753Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:04.0353284Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:04.0353709Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:04.0354106Z + shift 2024-08-20T22:21:04.0354405Z + for trap_add_name in "$@" 2024-08-20T22:21:04.0360568Z +++ trap -p EXIT 2024-08-20T22:21:04.0364369Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:04.0364760Z +++ extract_trap_cmd 2024-08-20T22:21:04.0365151Z +++ printf '%s\n' '' 2024-08-20T22:21:04.0365601Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:04.0368419Z + trap -- ' 2024-08-20T22:21:04.0368803Z cleanup_workspace' EXIT 2024-08-20T22:21:04.0369295Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:04.7481658Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:04.7504799Z + echo 'Environment variables:' 2024-08-20T22:21:04.7505219Z Environment variables: 2024-08-20T22:21:04.7505559Z + env 2024-08-20T22:21:04.7516316Z INSTALLED_DB=yes 2024-08-20T22:21:04.7517634Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.7518242Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.7518744Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.7519500Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.7520100Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.7520616Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.7521148Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.7521841Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.7522392Z HOSTNAME=b7a452073000 2024-08-20T22:21:04.7523362Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7524194Z GITHUB_ACTION=__self 2024-08-20T22:21:04.7524673Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.7528939Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.7532910Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.7533430Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.7533793Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.7534171Z TEST_CONFIG=dynamic_inductor_huggingface 2024-08-20T22:21:04.7534626Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.7535115Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.7535652Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.7536104Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.7537298Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.7537727Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.7538189Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.7538704Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.7539178Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.7539665Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.7540059Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.7540455Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7540909Z XLA_CUDA= 2024-08-20T22:21:04.7541420Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.7545170Z *** 2024-08-20T22:21:04.7545475Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.7545884Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.7546262Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.7546608Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.7547139Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.7547692Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.7548179Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7548704Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.7549196Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7550005Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.7550744Z UCC_HOME=/usr 2024-08-20T22:21:04.7551098Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.7551506Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.7551859Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.7552337Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.7552855Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.7553339Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.7553729Z SHARD_NUMBER=1 2024-08-20T22:21:04.7554051Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.7554409Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.7554787Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.7555237Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.7555731Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.7556220Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.7556601Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.7557039Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.7557502Z NUM_TEST_SHARDS=1 2024-08-20T22:21:04.7557808Z UCX_HOME=/usr 2024-08-20T22:21:04.7558345Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.7559460Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7560810Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.7562215Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7563355Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.7564033Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.7564378Z DASHBOARD_TAG= 2024-08-20T22:21:04.7564690Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.7565167Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.7565724Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.7566752Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7567677Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.7568097Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.7568484Z PR_NUMBER= 2024-08-20T22:21:04.7568774Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.7569131Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.7569585Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.7570060Z TERM=vt100 2024-08-20T22:21:04.7570403Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.7570830Z INSTALLED_VISION=yes 2024-08-20T22:21:04.7571154Z BRANCH= 2024-08-20T22:21:04.7571571Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.7571982Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.7572420Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.7573203Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.7574054Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.7574547Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.7575029Z REENABLED_ISSUES= 2024-08-20T22:21:04.7575341Z SHLVL=1 2024-08-20T22:21:04.7575604Z MAX_JOBS=14 2024-08-20T22:21:04.7575939Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.7576306Z NVARCH=x86_64 2024-08-20T22:21:04.7576617Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.7577083Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7577634Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.7578140Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.7578830Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.7579412Z GITHUB_JOB=test 2024-08-20T22:21:04.7579830Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.7580403Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.7580926Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.7581276Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.7581675Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.7582128Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.7582558Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.7582938Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.7583299Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.7583662Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.7584727Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.7585823Z GITHUB_BASE_REF= 2024-08-20T22:21:04.7586160Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.7586548Z CI=true 2024-08-20T22:21:04.7586879Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.7587317Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.7587703Z JOB_ID=29026528773 2024-08-20T22:21:04.7588021Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.7588366Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.7588687Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.7589140Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.7589616Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.7589964Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.7590340Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.7591279Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7592099Z NO_TD=False 2024-08-20T22:21:04.7592415Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.7592792Z _=/usr/bin/env 2024-08-20T22:21:04.7593282Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:04.7753180Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:04.7754397Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:04.7755450Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:04.7756401Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:04.7757015Z + BUILD_DIR=build 2024-08-20T22:21:04.7757355Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:04.7757757Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:04.7758105Z + SHARD_NUMBER=1 2024-08-20T22:21:04.7758411Z + NUM_TEST_SHARDS=1 2024-08-20T22:21:04.7758741Z + export VALGRIND=ON 2024-08-20T22:21:04.7759067Z + VALGRIND=ON 2024-08-20T22:21:04.7759550Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:04.7760149Z + [[ 0 == \1 ]] 2024-08-20T22:21:04.7760492Z + [[ False == \1 ]] 2024-08-20T22:21:04.7761037Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:04.7761988Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:04.7773456Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:04.7774248Z + [[ -n '' ]] 2024-08-20T22:21:04.7774623Z + echo 'Environment variables' 2024-08-20T22:21:04.7775204Z Environment variables 2024-08-20T22:21:04.7775535Z + env 2024-08-20T22:21:04.7782760Z INSTALLED_DB=yes 2024-08-20T22:21:04.7783689Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.7784616Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.7785413Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.7786264Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.7787100Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.7787796Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.7788506Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.7789174Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.7789690Z HOSTNAME=b7a452073000 2024-08-20T22:21:04.7790632Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7791462Z GITHUB_ACTION=__self 2024-08-20T22:21:04.7791827Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.7795697Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.7799687Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.7800219Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.7800851Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.7801245Z TEST_CONFIG=dynamic_inductor_huggingface 2024-08-20T22:21:04.7801796Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.7802295Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.7802778Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.7803237Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.7803664Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.7804066Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.7804533Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.7805052Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.7805595Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.7805943Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.7806356Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.7806789Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7807248Z XLA_CUDA= 2024-08-20T22:21:04.7807763Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.7808175Z *** 2024-08-20T22:21:04.7808475Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.7808894Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.7809283Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.7809630Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.7810150Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.7810713Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.7811202Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7811731Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.7812235Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7813056Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.7813798Z UCC_HOME=/usr 2024-08-20T22:21:04.7814146Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.7814759Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.7815121Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.7815601Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.7816114Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.7816714Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.7817101Z SHARD_NUMBER=1 2024-08-20T22:21:04.7817418Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.7817783Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.7818166Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.7818624Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.7819110Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.7819605Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.7819989Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.7820422Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.7820882Z NUM_TEST_SHARDS=1 2024-08-20T22:21:04.7821194Z UCX_HOME=/usr 2024-08-20T22:21:04.7821734Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.7822848Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7824194Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_huggingface, 1, 1, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.7825530Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7826679Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.7827347Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.7827690Z DASHBOARD_TAG= 2024-08-20T22:21:04.7828006Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.7828483Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.7829035Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.7830079Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7831010Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.7831430Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.7831813Z PR_NUMBER= 2024-08-20T22:21:04.7832108Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.7832437Z VALGRIND=ON 2024-08-20T22:21:04.7832753Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.7833204Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.7833678Z TERM=vt100 2024-08-20T22:21:04.7834025Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.7834448Z INSTALLED_VISION=yes 2024-08-20T22:21:04.7834771Z BRANCH= 2024-08-20T22:21:04.7835065Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.7835481Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.7835933Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.7836746Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.7837531Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.7838037Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.7838514Z REENABLED_ISSUES= 2024-08-20T22:21:04.7838820Z SHLVL=1 2024-08-20T22:21:04.7839100Z MAX_JOBS=14 2024-08-20T22:21:04.7839419Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.7839784Z NVARCH=x86_64 2024-08-20T22:21:04.7840098Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.7840567Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.7841113Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.7841714Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.7842394Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.7842982Z GITHUB_JOB=test 2024-08-20T22:21:04.7843396Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.7843967Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.7844497Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.7844841Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.7845329Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.7845789Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.7846275Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.7846647Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.7847006Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.7847373Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.7848513Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.7849582Z GITHUB_BASE_REF= 2024-08-20T22:21:04.7849919Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.7850302Z CI=true 2024-08-20T22:21:04.7850634Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.7851055Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.7851443Z JOB_ID=29026528773 2024-08-20T22:21:04.7851768Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.7852104Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.7852423Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.7852884Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.7853363Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.7853704Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.7854076Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.7855006Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_466bc1fa-1997-4fcf-9abe-924300075d60 2024-08-20T22:21:04.7855842Z NO_TD=False 2024-08-20T22:21:04.7856151Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.7856549Z _=/usr/bin/env 2024-08-20T22:21:04.7856930Z + echo 'Testing pytorch' 2024-08-20T22:21:04.7857278Z Testing pytorch 2024-08-20T22:21:04.7857614Z + export LANG=C.UTF-8 2024-08-20T22:21:04.7857964Z + LANG=C.UTF-8 2024-08-20T22:21:04.7858263Z + PR_NUMBER= 2024-08-20T22:21:04.7858647Z + [[ dynamic_inductor_huggingface == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:04.7859255Z + [[ dynamic_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:04.7859837Z + [[ dynamic_inductor_huggingface == \s\l\o\w ]] 2024-08-20T22:21:04.7860526Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:04.7861228Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:04.7861788Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.7862277Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.7862773Z + [[ dynamic_inductor_huggingface == *crossref* ]] 2024-08-20T22:21:04.7863390Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:04.7864044Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:04.7864707Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:04.7865286Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:04.7865833Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:05.5020819Z Collecting ninja==1.10.2 2024-08-20T22:21:05.5205372Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:05.5463614Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:06.1194152Z Installing collected packages: ninja 2024-08-20T22:21:06.1271746Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:06.1273086Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:06.1619281Z Successfully installed ninja-1.10.2 2024-08-20T22:21:06.2392572Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.2394845Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.2396313Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:06.2397748Z + install_tlparse 2024-08-20T22:21:06.2398175Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:06.2398747Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:06.6424188Z Collecting tlparse==0.3.25 2024-08-20T22:21:06.6616383Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:06.6999847Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:07.3221570Z Installing collected packages: tlparse 2024-08-20T22:21:07.3955188Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:07.4739406Z ++ python -m site --user-base 2024-08-20T22:21:07.4987470Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:07.4989665Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:07.4990376Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:07.4991051Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:07.4992025Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:07.4993209Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:07.4993930Z + cd test 2024-08-20T22:21:07.4994464Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:09.1410053Z + [[ dynamic_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:09.1410781Z + [[ dynamic_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:09.1413625Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:09.1414228Z + [[ dynamic_inductor_huggingface == *pr_time_benchmarks* ]] 2024-08-20T22:21:09.1415193Z + [[ dynamic_inductor_huggingface == *dynamo_eager* ]] 2024-08-20T22:21:09.1415941Z + [[ dynamic_inductor_huggingface == *aot_eager* ]] 2024-08-20T22:21:09.1416654Z + [[ dynamic_inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:21:09.1417231Z + [[ dynamic_inductor_huggingface == *inductor* ]] 2024-08-20T22:21:09.1417751Z + [[ dynamic_inductor_huggingface != *perf* ]] 2024-08-20T22:21:09.1418497Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:09.1418979Z + [[ dynamic_inductor_huggingface == *dynamic* ]] 2024-08-20T22:21:09.1419643Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-08-20T22:21:09.1420243Z + [[ dynamic_inductor_huggingface == *cpu* ]] 2024-08-20T22:21:09.1420772Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:09.1448275Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:09.1448977Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:09.1451539Z + cd test 2024-08-20T22:21:09.1452108Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:10.6241010Z PyTorch built with: 2024-08-20T22:21:10.6241974Z - GCC 9.4 2024-08-20T22:21:10.6242429Z - C++ Version: 201703 2024-08-20T22:21:10.6243328Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:10.6244438Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:10.6245143Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:10.6245681Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:10.6246185Z - NNPACK is enabled 2024-08-20T22:21:10.6246584Z - CPU capability usage: AVX2 2024-08-20T22:21:10.6247013Z - CUDA Runtime 12.1 2024-08-20T22:21:10.6247560Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:10.6248183Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:10.6248645Z - Magma 2.6.1 2024-08-20T22:21:10.6256703Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:10.6263422Z 2024-08-20T22:21:10.9547253Z + cd test 2024-08-20T22:21:10.9547991Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:12.2981248Z ATen/Parallel: 2024-08-20T22:21:12.2981809Z at::get_num_threads() : 8 2024-08-20T22:21:12.2982315Z at::get_num_interop_threads() : 16 2024-08-20T22:21:12.2982833Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:12.2983318Z omp_get_max_threads() : 8 2024-08-20T22:21:12.2984467Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:12.2985300Z mkl_get_max_threads() : 8 2024-08-20T22:21:12.2985941Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:12.2986593Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:12.2987030Z Environment variables: 2024-08-20T22:21:12.2987399Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:12.2987813Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:12.2988217Z ATen parallel backend: OpenMP 2024-08-20T22:21:12.2988480Z 2024-08-20T22:21:12.5799227Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:12.5800056Z + [[ dynamic_inductor_huggingface == *backward* ]] 2024-08-20T22:21:12.5801014Z + [[ dynamic_inductor_huggingface == *xla* ]] 2024-08-20T22:21:12.5801626Z + [[ dynamic_inductor_huggingface == *executorch* ]] 2024-08-20T22:21:12.5802204Z + [[ dynamic_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:12.5802923Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:12.5803498Z + [[ dynamic_inductor_huggingface == distributed ]] 2024-08-20T22:21:12.5804093Z + [[ dynamic_inductor_huggingface == *inductor_distributed* ]] 2024-08-20T22:21:12.5804768Z + [[ dynamic_inductor_huggingface == *inductor-halide* ]] 2024-08-20T22:21:12.5805467Z + [[ dynamic_inductor_huggingface == *inductor-micro-benchmark* ]] 2024-08-20T22:21:12.5806102Z + [[ dynamic_inductor_huggingface == *huggingface* ]] 2024-08-20T22:21:12.5806581Z + install_torchvision 2024-08-20T22:21:12.5806911Z + local orig_preload 2024-08-20T22:21:12.5807236Z + local commit 2024-08-20T22:21:12.5807548Z ++ get_pinned_commit vision 2024-08-20T22:21:12.5807952Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:12.5823232Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.5823788Z + orig_preload= 2024-08-20T22:21:12.5824142Z + '[' -n '' ']' 2024-08-20T22:21:12.5824984Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.5826445Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.9170687Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.9174641Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-xsne50e1 2024-08-20T22:21:12.9206177Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-xsne50e1 2024-08-20T22:21:14.5183701Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:14.5213258Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:15.8500696Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:16.2344732Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:18.7203874Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:18.7234754Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:18.7238895Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:18.7243574Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:18.7315073Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:18.7319449Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:18.7323283Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:18.7326499Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:18.7329652Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:18.7334708Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:18.7348891Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:18.7830443Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:18.7893240Z Building wheels for collected packages: torchvision 2024-08-20T22:22:37.0359872Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T22:22:37.0392464Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116162 sha256=b73a1f03127d7bb46b87149ba9904e179db910bef0e77f7f5796f310be170656 2024-08-20T22:22:37.0394146Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:37.0430872Z Successfully built torchvision 2024-08-20T22:22:37.5218511Z Installing collected packages: torchvision 2024-08-20T22:22:37.9407515Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:38.0848010Z + '[' -n '' ']' 2024-08-20T22:22:38.0848438Z + id=0 2024-08-20T22:22:38.0852641Z + test_dynamo_benchmark huggingface 0 2024-08-20T22:22:38.0853242Z ++ pwd 2024-08-20T22:22:38.0854370Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:38.0855001Z + local suite=huggingface 2024-08-20T22:22:38.0855666Z + shift 2024-08-20T22:22:38.0855946Z + local shard_id=0 2024-08-20T22:22:38.0856532Z + shift 2024-08-20T22:22:38.0857026Z + [[ dynamic_inductor_huggingface == *perf_compare* ]] 2024-08-20T22:22:38.0857773Z + [[ dynamic_inductor_huggingface == *perf* ]] 2024-08-20T22:22:38.0858722Z + [[ dynamic_inductor_huggingface == *cpu* ]] 2024-08-20T22:22:38.0859327Z + [[ dynamic_inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:22:38.0861492Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-08-20T22:22:38.0862154Z ++ pwd 2024-08-20T22:22:38.0862662Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:38.0863346Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:38.0887858Z + local name=inference 2024-08-20T22:22:38.0888362Z + shift 2024-08-20T22:22:38.0888757Z + local suite=huggingface 2024-08-20T22:22:38.0889202Z + shift 2024-08-20T22:22:38.0889562Z + local shard_id=0 2024-08-20T22:22:38.0889957Z + shift 2024-08-20T22:22:38.0890320Z + partition_flags=() 2024-08-20T22:22:38.0890769Z + local partition_flags 2024-08-20T22:22:38.0891318Z + [[ -n 1 ]] 2024-08-20T22:22:38.0891742Z + [[ -n 0 ]] 2024-08-20T22:22:38.0892528Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:22:38.0893288Z + [[ dynamic_inductor_huggingface == *perf_compare* ]] 2024-08-20T22:22:38.0893840Z + [[ dynamic_inductor_huggingface == *perf* ]] 2024-08-20T22:22:38.0894378Z + [[ dynamic_inductor_huggingface == *aot_inductor* ]] 2024-08-20T22:22:38.0894913Z + [[ dynamic_inductor_huggingface == *_avx2* ]] 2024-08-20T22:22:38.0895444Z + [[ dynamic_inductor_huggingface == *_avx512* ]] 2024-08-20T22:22:38.0897395Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-08-20T22:22:41.1517106Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:22:41.1518935Z warnings.warn( 2024-08-20T22:22:41.1954554Z 2024-08-20T22:22:41.1954978Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-08-20T22:51:05.8738905Z pass 2024-08-20T22:51:05.9068737Z TIMING: entire_frame_compile:26.29411 code_gen:14.30403 inductor_compile:22.47601 backend_compile:22.20929 total_wall_time:36.28871 2024-08-20T22:51:05.9070487Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:40791 | ProxyTorchDispatchMode.__torch_dispatch__:19979 | FakeTensor.__torch_dispatch__:7823 2024-08-20T22:51:05.9071672Z Dynamo produced 2 graphs covering 533 ops with 5 graph breaks (4 unique) 2024-08-20T22:51:11.1970428Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:51:11.1973355Z warnings.warn( 2024-08-20T22:51:11.3949520Z 2024-08-20T22:51:14.3443413Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:14.3443940Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:51:14.3444409Z cuda train AllenaiLongformerBase 2024-08-20T22:51:22.8353181Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T22:51:22.8354573Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T22:51:22.8355612Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] or: 2024-08-20T22:51:22.8356764Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T22:51:22.8358263Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph. 2024-08-20T22:51:22.8359252Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:51:22.8360161Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at: 2024-08-20T22:51:22.8361921Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:51:22.8363555Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] pred = mod(**cloned_inputs) 2024-08-20T22:51:22.8365558Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-08-20T22:51:22.8367187Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] outputs = self.longformer( 2024-08-20T22:51:22.8368968Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-08-20T22:51:22.8370599Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] encoder_outputs = self.encoder( 2024-08-20T22:51:22.8372394Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1291, in forward 2024-08-20T22:51:22.8374138Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] is_global_attn = is_index_global_attn.flatten().any().item() 2024-08-20T22:51:22.8375167Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:51:22.8375988Z W0820 22:51:22.834000 9808 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:52:45.2543397Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:52:45.2546723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in torch_dynamo_resume_in_forward_at_1291 2024-08-20T22:52:45.2547890Z layer_outputs = layer_module( 2024-08-20T22:52:45.2549287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-08-20T22:52:45.2550579Z self_attn_outputs = self.attention( 2024-08-20T22:52:45.2552122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-08-20T22:52:45.2553103Z self_outputs = self.self( 2024-08-20T22:52:45.2554251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 571, in forward 2024-08-20T22:52:45.2555684Z attn_scores = self._sliding_chunks_query_key_matmul( 2024-08-20T22:52:45.2557417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 836, in _sliding_chunks_query_key_matmul 2024-08-20T22:52:45.2558986Z query = self._chunk(query, window_overlap, getattr(self.config, "onnx_export", False)) 2024-08-20T22:52:45.2560326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 778, in _chunk 2024-08-20T22:52:45.2561551Z return hidden_states.as_strided(size=chunk_size, stride=chunk_stride) 2024-08-20T22:52:45.2562008Z 2024-08-20T22:52:45.7910076Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:52:45.7911734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1734, in forward 2024-08-20T22:52:45.7912739Z embedding_output = self.embeddings( 2024-08-20T22:52:45.7913852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 470, in forward 2024-08-20T22:52:45.7914872Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T22:52:45.7915224Z 2024-08-20T22:52:45.8334060Z W0820 22:52:45.832000 9808 torch/_logging/_internal.py:1058] [16/0] Profiler function will be ignored 2024-08-20T22:53:39.4978236Z pass 2024-08-20T22:53:39.5409295Z TIMING: entire_frame_compile:105.24036 inductor_compile:72.40893 backend_compile:84.90201 code_gen:41.83372 total_wall_time:129.12804 2024-08-20T22:53:39.5410781Z STATS: call_* op count: 2366 | FakeTensorMode.__torch_dispatch__:121873 | ProxyTorchDispatchMode.__torch_dispatch__:58081 | FakeTensor.__torch_dispatch__:20150 2024-08-20T22:53:39.5412511Z Dynamo produced 7 graphs covering 2366 ops with 9 graph breaks (5 unique) 2024-08-20T22:53:48.6290869Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:53:48.6292601Z warnings.warn( 2024-08-20T22:53:48.8424359Z 2024-08-20T22:53:52.8792142Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:52.8792655Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:53:52.8793107Z cuda train BartForCausalLM 2024-08-20T22:54:07.8623927Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:54:07.8625414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T22:54:07.8626390Z return super().forward(positions + self.offset) 2024-08-20T22:54:07.8626753Z 2024-08-20T22:54:13.6089516Z pass 2024-08-20T22:54:13.7375072Z TIMING: entire_frame_compile:4.88771 inductor_compile:4.12702 backend_compile:4.10344 code_gen:2.58042 total_wall_time:6.65064 2024-08-20T22:54:13.7377106Z STATS: call_* op count: 39 | FakeTensorMode.__torch_dispatch__:4032 | ProxyTorchDispatchMode.__torch_dispatch__:1679 | FakeTensor.__torch_dispatch__:746 2024-08-20T22:54:13.7378697Z Dynamo produced 5 graphs covering 39 ops with 6 graph breaks (5 unique) 2024-08-20T22:54:17.8733568Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:17.8735266Z warnings.warn( 2024-08-20T22:54:18.0803364Z 2024-08-20T22:54:25.3059064Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:25.3059714Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:54:25.3060363Z cuda train BartForConditionalGeneration 2024-08-20T22:54:50.7259659Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:54:50.7261911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T22:54:50.7262881Z return super().forward(positions + self.offset) 2024-08-20T22:54:50.7265430Z 2024-08-20T22:55:05.3091160Z pass 2024-08-20T22:55:05.5139061Z TIMING: entire_frame_compile:7.09933 inductor_compile:5.87253 backend_compile:5.4477 code_gen:3.54299 total_wall_time:9.45858 2024-08-20T22:55:05.5140494Z STATS: call_* op count: 93 | FakeTensorMode.__torch_dispatch__:8765 | ProxyTorchDispatchMode.__torch_dispatch__:3917 | FakeTensor.__torch_dispatch__:1628 2024-08-20T22:55:05.5141661Z Dynamo produced 7 graphs covering 93 ops with 8 graph breaks (5 unique) 2024-08-20T22:55:09.8334659Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:55:09.8336402Z warnings.warn( 2024-08-20T22:55:10.0310246Z 2024-08-20T22:55:12.1297432Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:55:12.1298044Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:55:12.1299687Z cuda train BertForMaskedLM 2024-08-20T22:55:48.3580280Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:55:48.3581600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:55:48.3582593Z pred = mod(**cloned_inputs) 2024-08-20T22:55:48.3585958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-08-20T22:55:48.3586908Z outputs = self.bert( 2024-08-20T22:55:48.3587874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-20T22:55:48.3588799Z embedding_output = self.embeddings( 2024-08-20T22:55:48.3589833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-20T22:55:48.3590788Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T22:55:48.3591143Z 2024-08-20T22:55:48.4947033Z W0820 22:55:48.493000 10838 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:56:34.4783511Z pass 2024-08-20T22:56:34.5413516Z TIMING: entire_frame_compile:66.17781 code_gen:24.30825 inductor_compile:42.78344 backend_compile:52.98531 total_wall_time:76.12935 2024-08-20T22:56:34.5414994Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:60797 | ProxyTorchDispatchMode.__torch_dispatch__:26954 | FakeTensor.__torch_dispatch__:16929 2024-08-20T22:56:34.5416177Z Dynamo produced 2 graphs covering 995 ops with 5 graph breaks (4 unique) 2024-08-20T22:56:41.9224900Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:56:41.9227742Z warnings.warn( 2024-08-20T22:56:42.1259216Z 2024-08-20T22:56:44.1084184Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:56:44.1084773Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:56:44.1085267Z cuda train BertForQuestionAnswering 2024-08-20T22:57:19.0066025Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:57:19.0069319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:57:19.0070342Z pred = mod(**cloned_inputs) 2024-08-20T22:57:19.0071599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1846, in forward 2024-08-20T22:57:19.0072746Z outputs = self.bert( 2024-08-20T22:57:19.0073852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-20T22:57:19.0074853Z embedding_output = self.embeddings( 2024-08-20T22:57:19.0076030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-20T22:57:19.0077071Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T22:57:19.0077431Z 2024-08-20T22:57:19.1370040Z W0820 22:57:19.136000 11158 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:58:02.1755307Z pass 2024-08-20T22:58:02.2411894Z TIMING: entire_frame_compile:62.88777 code_gen:21.53134 inductor_compile:38.689 backend_compile:48.99148 total_wall_time:72.2292 2024-08-20T22:58:02.2413341Z STATS: call_* op count: 993 | FakeTensorMode.__torch_dispatch__:60416 | ProxyTorchDispatchMode.__torch_dispatch__:26767 | FakeTensor.__torch_dispatch__:16739 2024-08-20T22:58:02.2414519Z Dynamo produced 2 graphs covering 993 ops with 5 graph breaks (4 unique) 2024-08-20T22:58:09.5994322Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:09.5996041Z warnings.warn( 2024-08-20T22:58:09.8065220Z 2024-08-20T22:58:38.9937448Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:38.9939688Z loading model: 0it [00:29, ?it/s] 2024-08-20T22:58:38.9940434Z cuda train BlenderbotForCausalLM 2024-08-20T22:58:38.9945612Z Traceback (most recent call last): 2024-08-20T22:58:38.9946381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T22:58:38.9950829Z self.model_iter_fn(model, example_inputs) 2024-08-20T22:58:38.9952009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T22:58:38.9952876Z pred = mod(**cloned_inputs) 2024-08-20T22:58:38.9954274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:38.9955491Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:38.9956774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:38.9957643Z return forward_call(*args, **kwargs) 2024-08-20T22:58:38.9958849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-08-20T22:58:38.9960071Z outputs = self.model.decoder( 2024-08-20T22:58:38.9961474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:38.9962693Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:38.9963946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:38.9964823Z return forward_call(*args, **kwargs) 2024-08-20T22:58:38.9965928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-08-20T22:58:38.9966909Z layer_outputs = decoder_layer( 2024-08-20T22:58:38.9967877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:38.9968795Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:38.9970859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:38.9971758Z return forward_call(*args, **kwargs) 2024-08-20T22:58:38.9972861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-08-20T22:58:38.9974162Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-08-20T22:58:38.9975333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:38.9976281Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:38.9977273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:38.9978123Z return forward_call(*args, **kwargs) 2024-08-20T22:58:38.9979218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-08-20T22:58:38.9980288Z query_states = self.q_proj(hidden_states) * self.scaling 2024-08-20T22:58:38.9981387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:58:38.9982296Z return self._call_impl(*args, **kwargs) 2024-08-20T22:58:38.9983256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:58:38.9984111Z return forward_call(*args, **kwargs) 2024-08-20T22:58:38.9985017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-08-20T22:58:38.9985891Z return F.linear(input, self.weight, self.bias) 2024-08-20T22:58:38.9989276Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 65562 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:58:38.9992063Z 2024-08-20T22:58:38.9992377Z The above exception was the direct cause of the following exception: 2024-08-20T22:58:38.9992832Z 2024-08-20T22:58:38.9992983Z Traceback (most recent call last): 2024-08-20T22:58:38.9993647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T22:58:38.9994318Z ) = runner.load_model( 2024-08-20T22:58:38.9995001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 454, in load_model 2024-08-20T22:58:38.9995777Z self.validate_model(model, example_inputs) 2024-08-20T22:58:38.9996556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T22:58:38.9997338Z raise RuntimeError("Eager run failed") from e 2024-08-20T22:58:38.9997824Z RuntimeError: Eager run failed 2024-08-20T22:58:38.9998085Z 2024-08-20T22:58:38.9998204Z eager_fail_to_run 2024-08-20T22:58:42.6838862Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:42.6840626Z warnings.warn( 2024-08-20T22:58:43.0127237Z 2024-08-20T22:58:44.5437346Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:44.5438235Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:58:44.5439180Z cuda train BlenderbotSmallForCausalLM 2024-08-20T22:58:44.5645251Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-08-20T22:58:53.3861010Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:58:53.3862627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-08-20T22:58:53.3863860Z return super().forward(positions) 2024-08-20T22:58:53.3864173Z 2024-08-20T22:58:57.6648487Z pass 2024-08-20T22:58:57.6914108Z TIMING: entire_frame_compile:5.94649 inductor_compile:4.75825 backend_compile:4.89354 code_gen:3.07495 total_wall_time:7.61401 2024-08-20T22:58:57.6915519Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:4665 | ProxyTorchDispatchMode.__torch_dispatch__:1916 | FakeTensor.__torch_dispatch__:792 2024-08-20T22:58:57.6916676Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T22:59:01.8891702Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:59:01.8893418Z warnings.warn( 2024-08-20T22:59:02.1150441Z 2024-08-20T22:59:04.1165282Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:59:04.1165758Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:59:04.1166292Z cuda train BlenderbotSmallForConditionalGeneration 2024-08-20T22:59:04.1641666Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:59:15.4691892Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T22:59:15.4693989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-08-20T22:59:15.4695409Z return super().forward(positions) 2024-08-20T22:59:15.4695822Z 2024-08-20T22:59:25.4773423Z pass 2024-08-20T22:59:25.5121365Z TIMING: entire_frame_compile:7.59773 inductor_compile:6.25879 backend_compile:5.98891 code_gen:3.08164 total_wall_time:10.13516 2024-08-20T22:59:25.5122805Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:9971 | ProxyTorchDispatchMode.__torch_dispatch__:4378 | FakeTensor.__torch_dispatch__:1692 2024-08-20T22:59:25.5123982Z Dynamo produced 7 graphs covering 121 ops with 8 graph breaks (5 unique) 2024-08-20T22:59:29.8339965Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:59:29.8341720Z warnings.warn( 2024-08-20T22:59:29.9903069Z 2024-08-20T22:59:32.1491792Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:59:32.1492414Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:59:32.1492951Z cuda train CamemBert 2024-08-20T23:00:08.1986560Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:00:08.1987772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:00:08.1989584Z pred = mod(**cloned_inputs) 2024-08-20T23:00:08.1990908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 979, in forward 2024-08-20T23:00:08.1991881Z outputs = self.roberta( 2024-08-20T23:00:08.1992927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 881, in forward 2024-08-20T23:00:08.1993908Z embedding_output = self.embeddings( 2024-08-20T23:00:08.1994992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 139, in forward 2024-08-20T23:00:08.1996007Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:00:08.1996373Z 2024-08-20T23:00:08.3362740Z W0820 23:00:08.335000 11835 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:00:53.9810778Z pass 2024-08-20T23:00:54.0498302Z TIMING: entire_frame_compile:65.70401 code_gen:24.74897 inductor_compile:42.20351 backend_compile:52.50259 total_wall_time:75.44882 2024-08-20T23:00:54.0500200Z STATS: call_* op count: 1003 | FakeTensorMode.__torch_dispatch__:60920 | ProxyTorchDispatchMode.__torch_dispatch__:27006 | FakeTensor.__torch_dispatch__:16957 2024-08-20T23:00:54.0504826Z Dynamo produced 2 graphs covering 1003 ops with 5 graph breaks (4 unique) 2024-08-20T23:01:01.4210109Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:01:01.4211875Z warnings.warn( 2024-08-20T23:01:01.6289988Z 2024-08-20T23:01:04.1720423Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:01:04.1720911Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:01:04.1721442Z cuda train DebertaForMaskedLM 2024-08-20T23:01:42.8112223Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:01:42.8114210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:01:42.8115864Z pred = mod(**cloned_inputs) 2024-08-20T23:01:42.8117770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1062, in forward 2024-08-20T23:01:42.8119304Z outputs = self.deberta( 2024-08-20T23:01:42.8120935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-08-20T23:01:42.8122622Z embedding_output = self.embeddings( 2024-08-20T23:01:42.8124362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-08-20T23:01:42.8125988Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:01:42.8126511Z 2024-08-20T23:01:42.9381973Z W0820 23:01:42.937000 12061 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:02:21.6624302Z pass 2024-08-20T23:02:21.7237314Z TIMING: entire_frame_compile:60.92734 code_gen:23.47815 inductor_compile:40.12574 backend_compile:48.65642 total_wall_time:71.14282 2024-08-20T23:02:21.7239377Z STATS: call_* op count: 1318 | FakeTensorMode.__torch_dispatch__:61543 | ProxyTorchDispatchMode.__torch_dispatch__:28174 | FakeTensor.__torch_dispatch__:15337 2024-08-20T23:02:21.7241093Z Dynamo produced 2 graphs covering 1318 ops with 5 graph breaks (4 unique) 2024-08-20T23:02:28.8348756Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:02:28.8350443Z warnings.warn( 2024-08-20T23:02:29.0079134Z 2024-08-20T23:02:31.1624881Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:02:31.1625393Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:02:31.1625876Z cuda train DebertaForQuestionAnswering 2024-08-20T23:03:07.8811593Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:03:07.8813371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:03:07.8814857Z pred = mod(**cloned_inputs) 2024-08-20T23:03:07.8816636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1388, in forward 2024-08-20T23:03:07.8818039Z outputs = self.deberta( 2024-08-20T23:03:07.8821043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-08-20T23:03:07.8822514Z embedding_output = self.embeddings( 2024-08-20T23:03:07.8824118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-08-20T23:03:07.8825900Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:03:07.8826431Z 2024-08-20T23:03:07.9998981Z W0820 23:03:07.999000 12399 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:03:44.6062376Z pass 2024-08-20T23:03:44.6732590Z TIMING: entire_frame_compile:57.78082 code_gen:21.60189 inductor_compile:37.56245 backend_compile:45.57688 total_wall_time:67.67796 2024-08-20T23:03:44.6736642Z STATS: call_* op count: 1316 | FakeTensorMode.__torch_dispatch__:61132 | ProxyTorchDispatchMode.__torch_dispatch__:27974 | FakeTensor.__torch_dispatch__:15143 2024-08-20T23:03:44.6737865Z Dynamo produced 2 graphs covering 1316 ops with 5 graph breaks (4 unique) 2024-08-20T23:03:51.7667132Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:03:51.7669058Z warnings.warn( 2024-08-20T23:03:51.9691190Z 2024-08-20T23:04:04.8566129Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:04.8566611Z loading model: 0it [00:12, ?it/s] 2024-08-20T23:04:04.8567159Z cuda train DebertaV2ForMaskedLM 2024-08-20T23:04:04.8568181Z pass_due_to_skip 2024-08-20T23:04:05.0827114Z TIMING: 2024-08-20T23:04:05.0827436Z STATS: call_* op count: 0 2024-08-20T23:04:05.0828006Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:04:08.7723508Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:04:08.7725426Z warnings.warn( 2024-08-20T23:04:08.9816170Z 2024-08-20T23:04:19.7486176Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:19.7486688Z loading model: 0it [00:10, ?it/s] 2024-08-20T23:04:19.7489407Z cuda train DebertaV2ForQuestionAnswering 2024-08-20T23:04:30.3972392Z ERROR:common: 2024-08-20T23:04:30.3972863Z Traceback (most recent call last): 2024-08-20T23:04:30.3973627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in check_accuracy 2024-08-20T23:04:30.3975148Z correct_result = self.run_n_iterations( 2024-08-20T23:04:30.3976164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-20T23:04:30.3977189Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-20T23:04:30.3978167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T23:04:30.3979300Z pred = mod(**cloned_inputs) 2024-08-20T23:04:30.3980804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.3981936Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.3983146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.3984188Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.3985536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-08-20T23:04:30.3986881Z outputs = self.deberta( 2024-08-20T23:04:30.3987876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.3989121Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.3990075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.3990922Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.3992148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-08-20T23:04:30.3993106Z encoder_outputs = self.encoder( 2024-08-20T23:04:30.3994096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.3994991Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.3995945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.3996790Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.3997867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-08-20T23:04:30.3998810Z output_states = layer_module( 2024-08-20T23:04:30.3999775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.4001730Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.4002793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.4003652Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.4004752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-08-20T23:04:30.4005731Z attention_output = self.attention( 2024-08-20T23:04:30.4006734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.4007650Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.4008608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.4009463Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.4010548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-08-20T23:04:30.4011499Z self_output = self.self( 2024-08-20T23:04:30.4012443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:04:30.4013346Z return self._call_impl(*args, **kwargs) 2024-08-20T23:04:30.4014303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:04:30.4015155Z return forward_call(*args, **kwargs) 2024-08-20T23:04:30.4016231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-08-20T23:04:30.4017435Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-08-20T23:04:30.4018515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply 2024-08-20T23:04:30.4019436Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:04:30.4020608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-08-20T23:04:30.4021598Z output = torch.softmax(output, self.dim) 2024-08-20T23:04:30.4025141Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 67051 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T23:04:30.4297496Z eager_1st_run_OOM 2024-08-20T23:04:30.4298665Z TIMING: 2024-08-20T23:04:30.4299340Z STATS: call_* op count: 0 2024-08-20T23:04:30.4299925Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:04:34.1301467Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:04:34.1303204Z warnings.warn( 2024-08-20T23:04:34.3607790Z 2024-08-20T23:04:35.8475911Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:35.8476420Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:04:35.8476888Z cuda train DistilBertForMaskedLM 2024-08-20T23:04:55.7480391Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:04:55.7481695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:04:55.7482697Z pred = mod(**cloned_inputs) 2024-08-20T23:04:55.7483984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-08-20T23:04:55.7484977Z dlbrt_output = self.distilbert( 2024-08-20T23:04:55.7486085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-08-20T23:04:55.7487272Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-08-20T23:04:55.7488586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-08-20T23:04:55.7489742Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-08-20T23:04:55.7490234Z 2024-08-20T23:04:55.8635147Z W0820 23:04:55.862000 12643 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:05:21.0456410Z pass 2024-08-20T23:05:21.0863623Z TIMING: entire_frame_compile:36.75618 code_gen:14.33636 inductor_compile:24.25029 backend_compile:29.72777 total_wall_time:41.8408 2024-08-20T23:05:21.0865094Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:32416 | ProxyTorchDispatchMode.__torch_dispatch__:14217 | FakeTensor.__torch_dispatch__:8794 2024-08-20T23:05:21.0867349Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-08-20T23:05:26.7524517Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:05:26.7526227Z warnings.warn( 2024-08-20T23:05:27.0220690Z 2024-08-20T23:05:28.2714103Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:05:28.2714690Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:05:28.2715287Z cuda train DistilBertForQuestionAnswering 2024-08-20T23:05:47.3762649Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:05:47.3763873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:05:47.3764871Z pred = mod(**cloned_inputs) 2024-08-20T23:05:47.3766386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1124, in forward 2024-08-20T23:05:47.3767514Z distilbert_output = self.distilbert( 2024-08-20T23:05:47.3769280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-08-20T23:05:47.3770468Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-08-20T23:05:47.3771763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-08-20T23:05:47.3773085Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-08-20T23:05:47.3773599Z 2024-08-20T23:05:47.4809668Z W0820 23:05:47.480000 13001 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:06:09.8736647Z pass 2024-08-20T23:06:09.9141570Z TIMING: entire_frame_compile:33.71116 code_gen:11.40434 inductor_compile:20.99822 backend_compile:26.75224 total_wall_time:38.40638 2024-08-20T23:06:09.9143171Z STATS: call_* op count: 539 | FakeTensorMode.__torch_dispatch__:32086 | ProxyTorchDispatchMode.__torch_dispatch__:14039 | FakeTensor.__torch_dispatch__:8611 2024-08-20T23:06:09.9144581Z Dynamo produced 2 graphs covering 539 ops with 5 graph breaks (4 unique) 2024-08-20T23:06:15.5188317Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:06:15.5190237Z warnings.warn( 2024-08-20T23:06:15.7057980Z 2024-08-20T23:06:17.8099884Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:06:17.8100639Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:06:17.8101108Z cuda train DistillGPT2 2024-08-20T23:06:37.6187709Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:06:37.6189894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:06:37.6190895Z pred = mod(**cloned_inputs) 2024-08-20T23:06:37.6192121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-08-20T23:06:37.6193064Z transformer_outputs = self.transformer( 2024-08-20T23:06:37.6194113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-08-20T23:06:37.6195032Z inputs_embeds = self.wte(input_ids) 2024-08-20T23:06:37.6195335Z 2024-08-20T23:06:37.7125621Z W0820 23:06:37.711000 13206 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:06:57.0376896Z pass 2024-08-20T23:06:57.0732302Z TIMING: entire_frame_compile:29.82859 code_gen:12.99136 inductor_compile:20.75452 backend_compile:24.00405 total_wall_time:35.26377 2024-08-20T23:06:57.0733877Z STATS: call_* op count: 571 | FakeTensorMode.__torch_dispatch__:26744 | ProxyTorchDispatchMode.__torch_dispatch__:11483 | FakeTensor.__torch_dispatch__:7064 2024-08-20T23:06:57.0735044Z Dynamo produced 2 graphs covering 571 ops with 5 graph breaks (4 unique) 2024-08-20T23:07:02.3643942Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:07:02.3645669Z warnings.warn( 2024-08-20T23:07:02.6368074Z 2024-08-20T23:07:02.6377375Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T23:07:03.4663760Z 2024-08-20T23:07:03.4664255Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:03.4666823Z cuda train ElectraForCausalLM 2024-08-20T23:07:38.2488847Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:07:38.2490627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:07:38.2495119Z pred = mod(**cloned_inputs) 2024-08-20T23:07:38.2496698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1617, in forward 2024-08-20T23:07:38.2498223Z outputs = self.electra( 2024-08-20T23:07:38.2499358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-08-20T23:07:38.2500579Z hidden_states = self.embeddings( 2024-08-20T23:07:38.2501767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-08-20T23:07:38.2502775Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:07:38.2503136Z 2024-08-20T23:07:39.0306918Z pass 2024-08-20T23:07:39.0496364Z TIMING: entire_frame_compile:22.96624 code_gen:9.80806 inductor_compile:17.20893 backend_compile:17.34614 total_wall_time:32.82681 2024-08-20T23:07:39.0497837Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:40391 | ProxyTorchDispatchMode.__torch_dispatch__:17991 | FakeTensor.__torch_dispatch__:6838 2024-08-20T23:07:39.0498999Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-08-20T23:07:44.4108086Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:07:44.4109793Z warnings.warn( 2024-08-20T23:07:44.6196550Z 2024-08-20T23:07:45.4312366Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:45.4312860Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:45.4313342Z cuda train ElectraForQuestionAnswering 2024-08-20T23:08:19.4191764Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:08:19.4193010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:08:19.4194012Z pred = mod(**cloned_inputs) 2024-08-20T23:08:19.4195268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1385, in forward 2024-08-20T23:08:19.4196277Z discriminator_hidden_states = self.electra( 2024-08-20T23:08:19.4197381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-08-20T23:08:19.4198334Z hidden_states = self.embeddings( 2024-08-20T23:08:19.4199373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-08-20T23:08:19.4200599Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:08:19.4201116Z 2024-08-20T23:08:19.5473846Z W0820 23:08:19.546000 13821 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:09:06.8150242Z pass 2024-08-20T23:09:06.8298854Z TIMING: entire_frame_compile:67.47194 code_gen:25.21618 inductor_compile:42.69428 backend_compile:53.35205 total_wall_time:76.9489 2024-08-20T23:09:06.8302056Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:60931 | ProxyTorchDispatchMode.__torch_dispatch__:26964 | FakeTensor.__torch_dispatch__:16890 2024-08-20T23:09:06.8304234Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T23:09:14.2340283Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:09:14.2341973Z warnings.warn( 2024-08-20T23:09:14.4449811Z 2024-08-20T23:09:16.9466497Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:09:16.9467677Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:09:16.9468199Z cuda train GPT2ForSequenceClassification 2024-08-20T23:09:54.2814794Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:09:54.2816255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1426, in forward 2024-08-20T23:09:54.2818611Z transformer_outputs = self.transformer( 2024-08-20T23:09:54.2819785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-08-20T23:09:54.2820797Z inputs_embeds = self.wte(input_ids) 2024-08-20T23:09:54.2821140Z 2024-08-20T23:09:54.3528785Z W0820 23:09:54.351000 14045 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:10:31.0204067Z pass 2024-08-20T23:10:31.0996767Z TIMING: entire_frame_compile:55.98245 inductor_compile:36.30941 backend_compile:43.39959 code_gen:22.11481 total_wall_time:65.36016 2024-08-20T23:10:31.0999306Z STATS: call_* op count: 1101 | FakeTensorMode.__torch_dispatch__:55263 | ProxyTorchDispatchMode.__torch_dispatch__:22507 | FakeTensor.__torch_dispatch__:13573 2024-08-20T23:10:31.1000701Z Dynamo produced 5 graphs covering 1101 ops with 7 graph breaks (5 unique) 2024-08-20T23:10:37.8588166Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:10:37.8589882Z warnings.warn( 2024-08-20T23:10:38.0397684Z 2024-08-20T23:10:39.5873940Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T23:10:39.7995552Z 2024-08-20T23:10:39.7995977Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:10:39.7998364Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T23:10:39.8007795Z cuda train GoogleFnet 2024-08-20T23:10:41.1398021Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T23:10:59.5122070Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:10:59.5123281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:10:59.5124299Z pred = mod(**cloned_inputs) 2024-08-20T23:10:59.5125506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 749, in forward 2024-08-20T23:10:59.5126396Z outputs = self.fnet( 2024-08-20T23:10:59.5127349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 581, in forward 2024-08-20T23:10:59.5128286Z embedding_output = self.embeddings( 2024-08-20T23:10:59.5129325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 148, in forward 2024-08-20T23:10:59.5130283Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:10:59.5130641Z 2024-08-20T23:10:59.6235877Z W0820 23:10:59.622000 14355 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:11:24.7409101Z pass 2024-08-20T23:11:24.7821959Z TIMING: entire_frame_compile:35.08491 code_gen:15.10131 inductor_compile:24.99091 backend_compile:27.82315 total_wall_time:40.79552 2024-08-20T23:11:24.7825750Z STATS: call_* op count: 573 | FakeTensorMode.__torch_dispatch__:27076 | ProxyTorchDispatchMode.__torch_dispatch__:12285 | FakeTensor.__torch_dispatch__:8814 2024-08-20T23:11:24.7826930Z Dynamo produced 2 graphs covering 573 ops with 5 graph breaks (4 unique) 2024-08-20T23:11:30.4837780Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:11:30.4839482Z warnings.warn( 2024-08-20T23:11:30.6615643Z 2024-08-20T23:11:32.8764659Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:11:32.8765801Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:11:32.8767085Z cuda train LayoutLMForMaskedLM 2024-08-20T23:12:09.8707916Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:12:09.8711258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:12:09.8712910Z pred = mod(**cloned_inputs) 2024-08-20T23:12:09.8714826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 938, in forward 2024-08-20T23:12:09.8716368Z outputs = self.layoutlm( 2024-08-20T23:12:09.8718032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-08-20T23:12:09.8719665Z embedding_output = self.embeddings( 2024-08-20T23:12:09.8721515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-08-20T23:12:09.8723168Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:12:09.8723731Z 2024-08-20T23:12:10.0102416Z W0820 23:12:10.009000 14675 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:12:56.8565934Z pass 2024-08-20T23:12:56.9216592Z TIMING: entire_frame_compile:67.30708 code_gen:26.45512 inductor_compile:44.21382 backend_compile:53.71774 total_wall_time:77.55628 2024-08-20T23:12:56.9218049Z STATS: call_* op count: 1033 | FakeTensorMode.__torch_dispatch__:62180 | ProxyTorchDispatchMode.__torch_dispatch__:27526 | FakeTensor.__torch_dispatch__:17218 2024-08-20T23:12:56.9219251Z Dynamo produced 2 graphs covering 1033 ops with 5 graph breaks (4 unique) 2024-08-20T23:13:04.3595253Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:13:04.3597007Z warnings.warn( 2024-08-20T23:13:04.5580392Z 2024-08-20T23:13:06.4932676Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:13:06.4933399Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:13:06.4934109Z cuda train LayoutLMForSequenceClassification 2024-08-20T23:13:44.4561386Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:13:44.4562917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 1060, in forward 2024-08-20T23:13:44.4563888Z outputs = self.layoutlm( 2024-08-20T23:13:44.4564947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-08-20T23:13:44.4566563Z embedding_output = self.embeddings( 2024-08-20T23:13:44.4567641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-08-20T23:13:44.4568651Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:13:44.4569012Z 2024-08-20T23:13:44.5951621Z W0820 23:13:44.594000 14877 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:14:32.4190072Z pass 2024-08-20T23:14:32.4878299Z TIMING: entire_frame_compile:68.85646 inductor_compile:42.19271 backend_compile:53.09333 code_gen:23.38474 total_wall_time:78.1303 2024-08-20T23:14:32.4879761Z STATS: call_* op count: 1030 | FakeTensorMode.__torch_dispatch__:66596 | ProxyTorchDispatchMode.__torch_dispatch__:27800 | FakeTensor.__torch_dispatch__:16804 2024-08-20T23:14:32.4881373Z Dynamo produced 5 graphs covering 1030 ops with 7 graph breaks (5 unique) 2024-08-20T23:14:39.9434178Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:14:39.9436414Z warnings.warn( 2024-08-20T23:14:40.1661847Z 2024-08-20T23:14:51.0144270Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:14:51.0144776Z loading model: 0it [00:10, ?it/s] 2024-08-20T23:14:51.0145256Z cuda train M2M100ForConditionalGeneration 2024-08-20T23:14:51.0921947Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:16:09.0434679Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:16:09.0435933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:16:09.0436927Z pred = mod(**cloned_inputs) 2024-08-20T23:16:09.0438129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1275, in forward 2024-08-20T23:16:09.0439059Z outputs = self.model( 2024-08-20T23:16:09.0440039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1162, in forward 2024-08-20T23:16:09.0440971Z encoder_outputs = self.encoder( 2024-08-20T23:16:09.0442078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 763, in forward 2024-08-20T23:16:09.0443130Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-08-20T23:16:09.0443566Z 2024-08-20T23:16:10.8486839Z pass 2024-08-20T23:16:10.9420578Z TIMING: entire_frame_compile:54.06156 code_gen:21.68545 inductor_compile:40.90828 backend_compile:41.29309 total_wall_time:75.10944 2024-08-20T23:16:10.9422077Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:101924 | ProxyTorchDispatchMode.__torch_dispatch__:47007 | FakeTensor.__torch_dispatch__:14711 2024-08-20T23:16:10.9423255Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-08-20T23:16:18.3278556Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:16:18.3280291Z warnings.warn( 2024-08-20T23:16:18.5077150Z 2024-08-20T23:16:22.5747958Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:16:22.5748726Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:16:22.5749413Z cuda train MBartForCausalLM 2024-08-20T23:16:22.6048782Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-08-20T23:16:30.0854152Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:16:30.0855596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T23:16:30.0856582Z return super().forward(positions + self.offset) 2024-08-20T23:16:30.0856951Z 2024-08-20T23:16:35.9935619Z pass 2024-08-20T23:16:36.0551950Z TIMING: entire_frame_compile:4.41745 inductor_compile:3.67303 backend_compile:3.55459 code_gen:1.92213 total_wall_time:5.70255 2024-08-20T23:16:36.0553353Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:4540 | ProxyTorchDispatchMode.__torch_dispatch__:1925 | FakeTensor.__torch_dispatch__:655 2024-08-20T23:16:36.0554487Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T23:16:40.2260201Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:16:40.2261934Z warnings.warn( 2024-08-20T23:16:40.4278648Z 2024-08-20T23:16:47.6171088Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:16:47.6171788Z loading model: 0it [00:07, ?it/s] 2024-08-20T23:16:47.6173080Z cuda train MBartForConditionalGeneration 2024-08-20T23:16:47.7158824Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:17:00.6751914Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:17:00.6754060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-08-20T23:17:00.6755038Z return super().forward(positions + self.offset) 2024-08-20T23:17:00.6755397Z 2024-08-20T23:17:15.9896144Z pass 2024-08-20T23:17:16.0955170Z TIMING: entire_frame_compile:7.66004 inductor_compile:6.29866 backend_compile:6.04027 code_gen:3.25633 total_wall_time:10.12114 2024-08-20T23:17:16.0956612Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:10227 | ProxyTorchDispatchMode.__torch_dispatch__:4496 | FakeTensor.__torch_dispatch__:1625 2024-08-20T23:17:16.0957772Z Dynamo produced 8 graphs covering 136 ops with 8 graph breaks (5 unique) 2024-08-20T23:17:20.3922886Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:17:20.3924604Z warnings.warn( 2024-08-20T23:17:20.6155394Z 2024-08-20T23:17:24.8339094Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:17:24.8339823Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:17:24.8340537Z cuda train MT5ForConditionalGeneration 2024-08-20T23:18:20.5596053Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:18:20.5597310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:18:20.5598314Z pred = mod(**cloned_inputs) 2024-08-20T23:18:20.5599518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1722, in forward 2024-08-20T23:18:20.5600758Z encoder_outputs = self.encoder( 2024-08-20T23:18:20.5602126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 989, in forward 2024-08-20T23:18:20.5603084Z inputs_embeds = self.embed_tokens(input_ids) 2024-08-20T23:18:20.5603435Z 2024-08-20T23:18:20.6916233Z W0820 23:18:20.690000 15747 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:19:08.0964399Z pass 2024-08-20T23:19:08.2398066Z TIMING: entire_frame_compile:79.91319 code_gen:28.67933 inductor_compile:52.62501 backend_compile:61.86224 total_wall_time:96.23248 2024-08-20T23:19:08.2399644Z STATS: call_* op count: 1763 | FakeTensorMode.__torch_dispatch__:97878 | ProxyTorchDispatchMode.__torch_dispatch__:44564 | FakeTensor.__torch_dispatch__:21452 2024-08-20T23:19:08.2401137Z Dynamo produced 2 graphs covering 1763 ops with 5 graph breaks (4 unique) 2024-08-20T23:19:16.3329371Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:19:16.3331552Z warnings.warn( 2024-08-20T23:19:16.5230548Z 2024-08-20T23:19:16.5268823Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T23:19:21.3653549Z 2024-08-20T23:19:21.3654203Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:19:21.3655163Z cuda train MegatronBertForCausalLM 2024-08-20T23:20:29.9250998Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:20:29.9252365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:20:29.9257207Z pred = mod(**cloned_inputs) 2024-08-20T23:20:29.9258737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1199, in forward 2024-08-20T23:20:29.9259831Z outputs = self.bert( 2024-08-20T23:20:29.9261113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-08-20T23:20:29.9262349Z embedding_output = self.embeddings( 2024-08-20T23:20:29.9263690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-08-20T23:20:29.9264759Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:20:29.9265114Z 2024-08-20T23:20:30.1126477Z W0820 23:20:30.111000 16144 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:22:04.0876021Z pass 2024-08-20T23:22:04.2888926Z TIMING: entire_frame_compile:130.51341 code_gen:50.74103 inductor_compile:84.9909 backend_compile:103.6552 total_wall_time:148.79296 2024-08-20T23:22:04.2891827Z STATS: call_* op count: 1922 | FakeTensorMode.__torch_dispatch__:118255 | ProxyTorchDispatchMode.__torch_dispatch__:52648 | FakeTensor.__torch_dispatch__:32930 2024-08-20T23:22:04.2894178Z Dynamo produced 2 graphs covering 1922 ops with 5 graph breaks (4 unique) 2024-08-20T23:22:15.2272303Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:22:15.2274063Z warnings.warn( 2024-08-20T23:22:15.4494611Z 2024-08-20T23:22:20.0236500Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:20.0236979Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:22:20.0239941Z cuda train MegatronBertForQuestionAnswering 2024-08-20T23:23:28.2943146Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:23:28.2944586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:23:28.2948476Z pred = mod(**cloned_inputs) 2024-08-20T23:23:28.2950129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1792, in forward 2024-08-20T23:23:28.2951292Z outputs = self.bert( 2024-08-20T23:23:28.2952450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-08-20T23:23:28.2953471Z embedding_output = self.embeddings( 2024-08-20T23:23:28.2954596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-08-20T23:23:28.2955658Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:23:28.2956013Z 2024-08-20T23:23:28.4724214Z W0820 23:23:28.471000 16560 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:24:55.8931607Z pass 2024-08-20T23:24:56.0949260Z TIMING: entire_frame_compile:124.30328 code_gen:44.11845 inductor_compile:77.62599 backend_compile:97.52723 total_wall_time:142.26999 2024-08-20T23:24:56.0950745Z STATS: call_* op count: 1916 | FakeTensorMode.__torch_dispatch__:117758 | ProxyTorchDispatchMode.__torch_dispatch__:52417 | FakeTensor.__torch_dispatch__:32732 2024-08-20T23:24:56.0952218Z Dynamo produced 2 graphs covering 1916 ops with 5 graph breaks (4 unique) 2024-08-20T23:25:07.0188308Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:25:07.0190986Z warnings.warn( 2024-08-20T23:25:07.3092413Z 2024-08-20T23:25:08.7771370Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:08.7771853Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:25:08.7772321Z cuda train MobileBertForMaskedLM 2024-08-20T23:27:12.7941501Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:27:12.7944270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:27:12.7945428Z pred = mod(**cloned_inputs) 2024-08-20T23:27:12.7946836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1089, in forward 2024-08-20T23:27:12.7948052Z outputs = self.mobilebert( 2024-08-20T23:27:12.7949431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-08-20T23:27:12.7950618Z embedding_output = self.embeddings( 2024-08-20T23:27:12.7951961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-08-20T23:27:12.7953053Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:27:12.7953478Z 2024-08-20T23:27:15.3960101Z pass 2024-08-20T23:27:15.4435147Z TIMING: entire_frame_compile:86.90216 code_gen:31.39082 inductor_compile:57.91262 backend_compile:59.77056 total_wall_time:120.11484 2024-08-20T23:27:15.4436827Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:158034 | ProxyTorchDispatchMode.__torch_dispatch__:74465 | FakeTensor.__torch_dispatch__:23011 2024-08-20T23:27:15.4438137Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-08-20T23:27:24.5221853Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:27:24.5223533Z warnings.warn( 2024-08-20T23:27:24.8214781Z 2024-08-20T23:27:26.1575147Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:27:26.1575629Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:27:26.1576126Z cuda train MobileBertForQuestionAnswering 2024-08-20T23:29:29.7326875Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:29:29.7329421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:29:29.7330436Z pred = mod(**cloned_inputs) 2024-08-20T23:29:29.7331756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1390, in forward 2024-08-20T23:29:29.7332763Z outputs = self.mobilebert( 2024-08-20T23:29:29.7333832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-08-20T23:29:29.7334849Z embedding_output = self.embeddings( 2024-08-20T23:29:29.7335944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-08-20T23:29:29.7336975Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:29:29.7337338Z 2024-08-20T23:29:32.3579513Z pass 2024-08-20T23:29:32.3914122Z TIMING: entire_frame_compile:86.74731 code_gen:30.87513 inductor_compile:57.53183 backend_compile:59.595 total_wall_time:119.70182 2024-08-20T23:29:32.3916015Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:157881 | ProxyTorchDispatchMode.__torch_dispatch__:74369 | FakeTensor.__torch_dispatch__:22989 2024-08-20T23:29:32.3917195Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-08-20T23:29:41.5799988Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:29:41.5802610Z warnings.warn( 2024-08-20T23:29:41.7774734Z 2024-08-20T23:29:44.4236966Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:44.4237633Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:29:44.4238210Z cuda train OPTForCausalLM 2024-08-20T23:29:44.4430031Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-08-20T23:29:53.5083780Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:29:53.5085232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 109, in forward 2024-08-20T23:29:53.5086214Z return super().forward(positions + self.offset) 2024-08-20T23:29:53.5086590Z 2024-08-20T23:29:59.4668677Z pass 2024-08-20T23:29:59.4922239Z TIMING: entire_frame_compile:4.92266 inductor_compile:4.64427 backend_compile:4.02996 code_gen:3.08023 total_wall_time:6.72146 2024-08-20T23:29:59.4923635Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:5003 | ProxyTorchDispatchMode.__torch_dispatch__:2038 | FakeTensor.__torch_dispatch__:802 2024-08-20T23:29:59.4924781Z Dynamo produced 6 graphs covering 82 ops with 6 graph breaks (5 unique) 2024-08-20T23:30:03.7053266Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:30:03.7055001Z warnings.warn( 2024-08-20T23:30:03.9297214Z 2024-08-20T23:30:05.8615420Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:05.8616005Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:30:05.8616517Z cuda train PLBartForCausalLM 2024-08-20T23:30:05.9785707Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-08-20T23:30:15.0191354Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:30:15.0194047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-08-20T23:30:15.0195362Z return super().forward(positions + self.offset) 2024-08-20T23:30:15.0195730Z 2024-08-20T23:30:18.3126156Z pass 2024-08-20T23:30:18.3411648Z TIMING: entire_frame_compile:5.64792 inductor_compile:4.87141 backend_compile:4.7768 code_gen:2.62858 total_wall_time:7.62705 2024-08-20T23:30:18.3413093Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:4751 | ProxyTorchDispatchMode.__torch_dispatch__:1942 | FakeTensor.__torch_dispatch__:796 2024-08-20T23:30:18.3414227Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T23:30:22.5448690Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:30:22.5450477Z warnings.warn( 2024-08-20T23:30:22.7909046Z 2024-08-20T23:30:26.3012116Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:26.3012615Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:30:26.3013103Z cuda train PLBartForConditionalGeneration 2024-08-20T23:30:26.3551483Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:30:38.5477058Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:30:38.5479197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-08-20T23:30:38.5480203Z return super().forward(positions + self.offset) 2024-08-20T23:30:38.5480740Z 2024-08-20T23:30:47.2620331Z pass 2024-08-20T23:30:47.3007030Z TIMING: entire_frame_compile:8.02127 inductor_compile:7.0946 backend_compile:6.3683 code_gen:3.68991 total_wall_time:11.23749 2024-08-20T23:30:47.3009867Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:10524 | ProxyTorchDispatchMode.__torch_dispatch__:4595 | FakeTensor.__torch_dispatch__:1763 2024-08-20T23:30:47.3012148Z Dynamo produced 8 graphs covering 137 ops with 8 graph breaks (5 unique) 2024-08-20T23:30:51.6595006Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:30:51.6596862Z warnings.warn( 2024-08-20T23:30:51.8552174Z 2024-08-20T23:30:58.1209543Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:30:58.1210061Z loading model: 0it [00:06, ?it/s] 2024-08-20T23:30:58.1210600Z cuda train PegasusForCausalLM 2024-08-20T23:30:58.1511814Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-08-20T23:31:12.7600619Z pass 2024-08-20T23:31:12.8351345Z TIMING: entire_frame_compile:5.38636 inductor_compile:4.56876 backend_compile:4.54777 code_gen:2.31731 total_wall_time:7.2506 2024-08-20T23:31:12.8352763Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:4616 | ProxyTorchDispatchMode.__torch_dispatch__:1907 | FakeTensor.__torch_dispatch__:793 2024-08-20T23:31:12.8353891Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T23:31:17.0448514Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:31:17.0450228Z warnings.warn( 2024-08-20T23:31:17.2582141Z 2024-08-20T23:31:27.6597448Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:27.6598278Z loading model: 0it [00:10, ?it/s] 2024-08-20T23:31:27.6598925Z cuda train PegasusForConditionalGeneration 2024-08-20T23:31:27.7342444Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T23:31:54.6242252Z pass 2024-08-20T23:31:54.7520912Z TIMING: entire_frame_compile:7.37406 inductor_compile:6.25115 backend_compile:5.79173 code_gen:2.95753 total_wall_time:10.11689 2024-08-20T23:31:54.7523854Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:9942 | ProxyTorchDispatchMode.__torch_dispatch__:4369 | FakeTensor.__torch_dispatch__:1731 2024-08-20T23:31:54.7525041Z Dynamo produced 7 graphs covering 121 ops with 7 graph breaks (4 unique) 2024-08-20T23:31:59.1323328Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:31:59.1325090Z warnings.warn( 2024-08-20T23:31:59.3187913Z 2024-08-20T23:31:59.3216910Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-08-20T23:32:01.5821506Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T23:32:01.5825656Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T23:32:01.8642819Z 2024-08-20T23:32:01.8643575Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:32:01.8644121Z cuda train RobertaForCausalLM 2024-08-20T23:32:37.4518209Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:32:37.4520018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:32:37.4521891Z pred = mod(**cloned_inputs) 2024-08-20T23:32:37.4523329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 953, in forward 2024-08-20T23:32:37.4524328Z outputs = self.roberta( 2024-08-20T23:32:37.4525345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-08-20T23:32:37.4526343Z embedding_output = self.embeddings( 2024-08-20T23:32:37.4527396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-08-20T23:32:37.4528382Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:32:37.4528743Z 2024-08-20T23:32:37.5898189Z W0820 23:32:37.589000 18224 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:33:23.0992772Z pass 2024-08-20T23:33:23.1527654Z TIMING: entire_frame_compile:65.01624 code_gen:23.96625 inductor_compile:42.32271 backend_compile:51.85626 total_wall_time:74.74435 2024-08-20T23:33:23.1529271Z STATS: call_* op count: 1007 | FakeTensorMode.__torch_dispatch__:61027 | ProxyTorchDispatchMode.__torch_dispatch__:27046 | FakeTensor.__torch_dispatch__:16964 2024-08-20T23:33:23.1530441Z Dynamo produced 2 graphs covering 1007 ops with 5 graph breaks (4 unique) 2024-08-20T23:33:30.5297069Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:33:30.5298795Z warnings.warn( 2024-08-20T23:33:30.7088572Z 2024-08-20T23:33:32.4678148Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T23:33:32.4680304Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T23:33:32.7877672Z 2024-08-20T23:33:32.7878345Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:33:32.7878905Z cuda train RobertaForQuestionAnswering 2024-08-20T23:34:07.3965542Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:34:07.3966821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:34:07.3972367Z pred = mod(**cloned_inputs) 2024-08-20T23:34:07.3973916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1500, in forward 2024-08-20T23:34:07.3975086Z outputs = self.roberta( 2024-08-20T23:34:07.3976195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-08-20T23:34:07.3977246Z embedding_output = self.embeddings( 2024-08-20T23:34:07.3978444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-08-20T23:34:07.3979503Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:34:07.3979877Z 2024-08-20T23:34:07.5263070Z W0820 23:34:07.525000 18402 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:34:50.1059670Z pass 2024-08-20T23:34:50.1702761Z TIMING: entire_frame_compile:61.98053 code_gen:21.03489 inductor_compile:38.2337 backend_compile:47.9829 total_wall_time:71.41241 2024-08-20T23:34:50.1706592Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:60506 | ProxyTorchDispatchMode.__torch_dispatch__:26802 | FakeTensor.__torch_dispatch__:16758 2024-08-20T23:34:50.1707779Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T23:34:57.5265842Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:34:57.5267736Z warnings.warn( 2024-08-20T23:34:57.7614442Z 2024-08-20T23:34:58.4086394Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:34:58.4086912Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:34:58.4089883Z cuda train Speech2Text2ForCausalLM 2024-08-20T23:34:58.5241306Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-08-20T23:35:09.4235611Z pass 2024-08-20T23:35:09.4286070Z TIMING: entire_frame_compile:4.94042 inductor_compile:4.52744 backend_compile:4.07891 code_gen:2.3616 total_wall_time:6.51749 2024-08-20T23:35:09.4287500Z STATS: call_* op count: 68 | FakeTensorMode.__torch_dispatch__:4770 | ProxyTorchDispatchMode.__torch_dispatch__:1940 | FakeTensor.__torch_dispatch__:739 2024-08-20T23:35:09.4288650Z Dynamo produced 6 graphs covering 68 ops with 6 graph breaks (5 unique) 2024-08-20T23:35:13.6130831Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:35:13.6134006Z warnings.warn( 2024-08-20T23:35:13.8959806Z 2024-08-20T23:35:15.4454021Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:35:15.4454557Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:35:15.4455071Z cuda train T5ForConditionalGeneration 2024-08-20T23:35:56.1484240Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:35:56.1485478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:35:56.1489101Z pred = mod(**cloned_inputs) 2024-08-20T23:35:56.1492000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-08-20T23:35:56.1492938Z encoder_outputs = self.encoder( 2024-08-20T23:35:56.1493938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-08-20T23:35:56.1494875Z inputs_embeds = self.embed_tokens(input_ids) 2024-08-20T23:35:56.1495225Z 2024-08-20T23:35:56.2553954Z W0820 23:35:56.254000 18782 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:36:29.8992937Z pass 2024-08-20T23:36:29.9601732Z TIMING: entire_frame_compile:57.13139 code_gen:20.52053 inductor_compile:35.73359 backend_compile:43.92118 total_wall_time:67.40352 2024-08-20T23:36:29.9603871Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:67817 | ProxyTorchDispatchMode.__torch_dispatch__:30411 | FakeTensor.__torch_dispatch__:14024 2024-08-20T23:36:29.9605621Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T23:36:36.6627068Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:36:36.6628903Z warnings.warn( 2024-08-20T23:36:36.8689230Z 2024-08-20T23:36:38.3955864Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:36:38.3956806Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:36:38.3958322Z cuda train T5Small 2024-08-20T23:37:03.3681292Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:37:03.3682762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:37:03.3687290Z pred = mod(**cloned_inputs) 2024-08-20T23:37:03.3688808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-08-20T23:37:03.3689728Z encoder_outputs = self.encoder( 2024-08-20T23:37:03.3690708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-08-20T23:37:03.3691662Z inputs_embeds = self.embed_tokens(input_ids) 2024-08-20T23:37:03.3692015Z 2024-08-20T23:37:03.4507077Z W0820 23:37:03.449000 19134 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:37:20.1871296Z pass 2024-08-20T23:37:20.2447553Z TIMING: entire_frame_compile:34.32572 inductor_compile:5.14127 backend_compile:22.01059 total_wall_time:35.64829 2024-08-20T23:37:20.2449060Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:55256 | ProxyTorchDispatchMode.__torch_dispatch__:30411 | FakeTensor.__torch_dispatch__:2452 2024-08-20T23:37:20.2450239Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T23:37:24.8992311Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:37:24.8994005Z warnings.warn( 2024-08-20T23:37:25.0824120Z 2024-08-20T23:37:29.1405834Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:29.1406439Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:37:29.1406956Z cuda train TrOCRForCausalLM 2024-08-20T23:37:29.1714653Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-08-20T23:37:38.0964655Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:37:38.0966292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 66, in forward 2024-08-20T23:37:38.0967269Z return super().forward(positions + self.offset) 2024-08-20T23:37:38.0967634Z 2024-08-20T23:37:44.0633772Z pass 2024-08-20T23:37:44.1434016Z TIMING: entire_frame_compile:5.54704 inductor_compile:4.41486 backend_compile:4.65641 code_gen:2.87196 total_wall_time:7.33614 2024-08-20T23:37:44.1437006Z STATS: call_* op count: 59 | FakeTensorMode.__torch_dispatch__:4709 | ProxyTorchDispatchMode.__torch_dispatch__:1925 | FakeTensor.__torch_dispatch__:796 2024-08-20T23:37:44.1439111Z Dynamo produced 6 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-08-20T23:37:48.2941150Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:37:48.2942933Z warnings.warn( 2024-08-20T23:37:48.5651812Z 2024-08-20T23:37:59.8236734Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:37:59.8237349Z loading model: 0it [00:11, ?it/s] 2024-08-20T23:37:59.8241760Z cuda train XGLMForCausalLM 2024-08-20T23:37:59.8777403Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-08-20T23:38:04.1294329Z skipping cudagraphs due to mutated inputs (1 instances). Found from : 2024-08-20T23:38:04.1295832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 174, in forward 2024-08-20T23:38:04.1296754Z position_ids += self.offset 2024-08-20T23:38:04.1297230Z 2024-08-20T23:38:20.6570121Z pass 2024-08-20T23:38:20.7457869Z TIMING: entire_frame_compile:5.19635 inductor_compile:4.66354 backend_compile:4.15723 code_gen:2.55031 total_wall_time:6.90684 2024-08-20T23:38:20.7460310Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:4948 | ProxyTorchDispatchMode.__torch_dispatch__:2016 | FakeTensor.__torch_dispatch__:835 2024-08-20T23:38:20.7461443Z Dynamo produced 6 graphs covering 67 ops with 6 graph breaks (5 unique) 2024-08-20T23:38:24.9874784Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:38:24.9876491Z warnings.warn( 2024-08-20T23:38:25.1842979Z 2024-08-20T23:38:29.8658193Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:38:29.8658677Z loading model: 0it [00:04, ?it/s] 2024-08-20T23:38:29.8659187Z cuda train XLNetLMHeadModel 2024-08-20T23:39:20.1226652Z W0820 23:39:20.121000 19713 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-20T23:39:37.8394298Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-08-20T23:39:37.8395651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:39:37.8396625Z pred = mod(**cloned_inputs) 2024-08-20T23:39:37.8397820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1446, in forward 2024-08-20T23:39:37.8398797Z transformer_outputs = self.transformer( 2024-08-20T23:39:37.8399849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1203, in forward 2024-08-20T23:39:37.8401321Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2024-08-20T23:39:37.8402629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1055, in relative_positional_encoding 2024-08-20T23:39:37.8403966Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2024-08-20T23:39:37.8405345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1014, in positional_embedding 2024-08-20T23:39:37.8406405Z pos_emb = pos_emb[:, None, :] 2024-08-20T23:39:37.8406748Z 2024-08-20T23:40:02.9639083Z W0820 23:40:02.962000 19713 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:41:32.8088393Z pass 2024-08-20T23:41:33.0206070Z TIMING: entire_frame_compile:146.36922 code_gen:52.62206 inductor_compile:92.75382 backend_compile:115.86204 total_wall_time:168.9385 2024-08-20T23:41:33.0208993Z STATS: call_* op count: 1873 | FakeTensorMode.__torch_dispatch__:162744 | ProxyTorchDispatchMode.__torch_dispatch__:66725 | FakeTensor.__torch_dispatch__:33630 2024-08-20T23:41:33.0210309Z Dynamo produced 2 graphs covering 1873 ops with 5 graph breaks (4 unique) 2024-08-20T23:41:44.2078771Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:41:44.2080686Z warnings.warn( 2024-08-20T23:41:44.3936747Z 2024-08-20T23:41:46.7274170Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:41:46.7274888Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:41:46.7278250Z cuda train YituTechConvBert 2024-08-20T23:42:41.0760345Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:42:41.0763384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T23:42:41.0764960Z pred = mod(**cloned_inputs) 2024-08-20T23:42:41.0766314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 918, in forward 2024-08-20T23:42:41.0767432Z generator_hidden_states = self.convbert( 2024-08-20T23:42:41.0768683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 834, in forward 2024-08-20T23:42:41.0769719Z hidden_states = self.embeddings( 2024-08-20T23:42:41.0770909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 230, in forward 2024-08-20T23:42:41.0772025Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-20T23:42:41.0772387Z 2024-08-20T23:42:41.2446297Z W0820 23:42:41.243000 20139 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:43:48.8236796Z pass 2024-08-20T23:43:48.8784207Z TIMING: entire_frame_compile:98.93498 code_gen:37.43767 inductor_compile:63.80006 backend_compile:78.20985 total_wall_time:114.26045 2024-08-20T23:43:48.8786430Z STATS: call_* op count: 1511 | FakeTensorMode.__torch_dispatch__:90845 | ProxyTorchDispatchMode.__torch_dispatch__:39917 | FakeTensor.__torch_dispatch__:24103 2024-08-20T23:43:48.8788184Z Dynamo produced 2 graphs covering 1511 ops with 5 graph breaks (4 unique) 2024-08-20T23:43:55.1606638Z accuracy pass_rate=93.48% 2024-08-20T23:43:55.1610593Z calls_captured gmean=0.00x mean=750.522x 2024-08-20T23:43:55.1614254Z unique_graphs gmean=0.00x mean=3.348x 2024-08-20T23:43:55.1617865Z graph_breaks gmean=0.00x mean=5.196x 2024-08-20T23:43:55.1621393Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-08-20T23:43:55.1625065Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:43:55.1628552Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:43:55.1632598Z cudagraph_skips gmean=0.00x mean=0.891x 2024-08-20T23:43:56.3243601Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_huggingface_training.csv 2024-08-20T23:43:56.5877932Z AlbertForMaskedLM PASS 2024-08-20T23:43:56.5882426Z AlbertForQuestionAnswering PASS 2024-08-20T23:43:56.5887541Z AllenaiLongformerBase PASS 2024-08-20T23:43:56.5892305Z BartForCausalLM PASS 2024-08-20T23:43:56.5897351Z BartForConditionalGeneration PASS 2024-08-20T23:43:56.5902408Z BertForMaskedLM PASS 2024-08-20T23:43:56.5907434Z BertForQuestionAnswering PASS 2024-08-20T23:43:56.5912058Z BlenderbotForCausalLM XFAIL 2024-08-20T23:43:56.5917471Z BlenderbotSmallForCausalLM PASS 2024-08-20T23:43:56.5922777Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T23:43:56.5927616Z CamemBert PASS 2024-08-20T23:43:56.5932413Z DebertaForMaskedLM PASS 2024-08-20T23:43:56.5937199Z DebertaForQuestionAnswering PASS 2024-08-20T23:43:56.5941715Z DebertaV2ForMaskedLM XFAIL 2024-08-20T23:43:56.5946191Z DebertaV2ForQuestionAnswering XFAIL 2024-08-20T23:43:56.5950990Z DistilBertForMaskedLM PASS 2024-08-20T23:43:56.5955531Z DistilBertForQuestionAnswering PASS 2024-08-20T23:43:56.5959913Z DistillGPT2 PASS 2024-08-20T23:43:56.5964766Z ElectraForCausalLM PASS 2024-08-20T23:43:56.5969266Z ElectraForQuestionAnswering PASS 2024-08-20T23:43:56.5973703Z GPT2ForSequenceClassification PASS 2024-08-20T23:43:56.5978647Z GoogleFnet PASS 2024-08-20T23:43:56.5982866Z LayoutLMForMaskedLM PASS 2024-08-20T23:43:56.5987336Z LayoutLMForSequenceClassification PASS 2024-08-20T23:43:56.5991863Z M2M100ForConditionalGeneration PASS 2024-08-20T23:43:56.5996287Z MBartForCausalLM PASS 2024-08-20T23:43:56.6001080Z MBartForConditionalGeneration PASS 2024-08-20T23:43:56.6006102Z MT5ForConditionalGeneration PASS 2024-08-20T23:43:56.6010564Z MegatronBertForCausalLM PASS 2024-08-20T23:43:56.6015090Z MegatronBertForQuestionAnswering PASS 2024-08-20T23:43:56.6019467Z MobileBertForMaskedLM PASS 2024-08-20T23:43:56.6023985Z MobileBertForQuestionAnswering PASS 2024-08-20T23:43:56.6028601Z OPTForCausalLM PASS 2024-08-20T23:43:56.6033160Z PLBartForCausalLM PASS 2024-08-20T23:43:56.6037654Z PLBartForConditionalGeneration PASS 2024-08-20T23:43:56.6042039Z PegasusForCausalLM PASS 2024-08-20T23:43:56.6046925Z PegasusForConditionalGeneration PASS 2024-08-20T23:43:56.6051326Z RobertaForCausalLM PASS 2024-08-20T23:43:56.6055859Z RobertaForQuestionAnswering PASS 2024-08-20T23:43:56.6060271Z Speech2Text2ForCausalLM PASS 2024-08-20T23:43:56.6064834Z T5ForConditionalGeneration PASS 2024-08-20T23:43:56.6069257Z T5Small PASS 2024-08-20T23:43:56.6073828Z TrOCRForCausalLM PASS 2024-08-20T23:43:56.6078364Z XGLMForCausalLM PASS 2024-08-20T23:43:56.6082984Z XLNetLMHeadModel PASS 2024-08-20T23:43:56.6087497Z YituTechConvBert PASS 2024-08-20T23:43:56.6521154Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_huggingface_training.csv 2024-08-20T23:43:56.9148998Z AlbertForMaskedLM PASS 2024-08-20T23:43:56.9154562Z AlbertForQuestionAnswering PASS 2024-08-20T23:43:56.9158932Z AllenaiLongformerBase PASS 2024-08-20T23:43:56.9162507Z BartForCausalLM PASS 2024-08-20T23:43:56.9167505Z BartForConditionalGeneration PASS 2024-08-20T23:43:56.9171813Z BertForMaskedLM PASS 2024-08-20T23:43:56.9176442Z BertForQuestionAnswering PASS 2024-08-20T23:43:56.9180909Z BlenderbotForCausalLM PASS 2024-08-20T23:43:56.9185557Z BlenderbotSmallForCausalLM PASS 2024-08-20T23:43:56.9190173Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T23:43:56.9194715Z CamemBert PASS 2024-08-20T23:43:56.9199317Z DebertaForMaskedLM PASS 2024-08-20T23:43:56.9204643Z DebertaForQuestionAnswering PASS 2024-08-20T23:43:56.9209135Z DebertaV2ForMaskedLM PASS 2024-08-20T23:43:56.9213618Z DebertaV2ForQuestionAnswering PASS 2024-08-20T23:43:56.9218191Z DistilBertForMaskedLM PASS 2024-08-20T23:43:56.9222800Z DistilBertForQuestionAnswering PASS 2024-08-20T23:43:56.9227263Z DistillGPT2 PASS 2024-08-20T23:43:56.9231919Z ElectraForCausalLM PASS 2024-08-20T23:43:56.9236445Z ElectraForQuestionAnswering PASS 2024-08-20T23:43:56.9241035Z GPT2ForSequenceClassification PASS 2024-08-20T23:43:56.9245700Z GoogleFnet PASS 2024-08-20T23:43:56.9250220Z LayoutLMForMaskedLM PASS 2024-08-20T23:43:56.9254844Z LayoutLMForSequenceClassification PASS 2024-08-20T23:43:56.9259393Z M2M100ForConditionalGeneration PASS 2024-08-20T23:43:56.9263947Z MBartForCausalLM PASS 2024-08-20T23:43:56.9268684Z MBartForConditionalGeneration PASS 2024-08-20T23:43:56.9273089Z MT5ForConditionalGeneration PASS 2024-08-20T23:43:56.9277661Z MegatronBertForCausalLM PASS 2024-08-20T23:43:56.9282262Z MegatronBertForQuestionAnswering PASS 2024-08-20T23:43:56.9287083Z MobileBertForMaskedLM PASS 2024-08-20T23:43:56.9291437Z MobileBertForQuestionAnswering PASS 2024-08-20T23:43:56.9296013Z OPTForCausalLM PASS 2024-08-20T23:43:56.9300701Z PLBartForCausalLM PASS 2024-08-20T23:43:56.9305650Z PLBartForConditionalGeneration PASS 2024-08-20T23:43:56.9310130Z PegasusForCausalLM PASS 2024-08-20T23:43:56.9314722Z PegasusForConditionalGeneration PASS 2024-08-20T23:43:56.9319239Z RobertaForCausalLM PASS 2024-08-20T23:43:56.9324190Z RobertaForQuestionAnswering PASS 2024-08-20T23:43:56.9328637Z Speech2Text2ForCausalLM PASS 2024-08-20T23:43:56.9333326Z T5ForConditionalGeneration PASS 2024-08-20T23:43:56.9337660Z T5Small PASS 2024-08-20T23:43:56.9342183Z TrOCRForCausalLM PASS 2024-08-20T23:43:56.9346659Z XGLMForCausalLM PASS 2024-08-20T23:43:56.9351229Z XLNetLMHeadModel PASS 2024-08-20T23:43:56.9355803Z YituTechConvBert PASS 2024-08-20T23:43:56.9788773Z + cleanup_workspace 2024-08-20T23:43:56.9789912Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:43:56.9791121Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:43:56.9792345Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:43:56.9793184Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:43:56.9794333Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:43:56.9795311Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:43:56.9796026Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:43:57.7135012Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:43:57.7135546Z cat test/**/*_toprint.log || true 2024-08-20T23:43:57.7149230Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:57.7149745Z env: 2024-08-20T23:43:57.7150028Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:57.7150499Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:57.7151275Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:57.7151930Z ##[endgroup] 2024-08-20T23:43:57.7252753Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:43:57.7285494Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:43:57.7285972Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:43:57.7294069Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:57.7294579Z env: 2024-08-20T23:43:57.7294870Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:57.7295348Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:57.7296120Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:57.7296803Z MONITOR_SCRIPT_PID: 52535 2024-08-20T23:43:57.7297179Z ##[endgroup] 2024-08-20T23:43:57.7498750Z Prepare all required actions 2024-08-20T23:43:57.7499199Z Getting action download info 2024-08-20T23:43:57.9863468Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:43:58.1509274Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:43:58.1509765Z with: 2024-08-20T23:43:58.1510411Z file-suffix: test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773 2024-08-20T23:43:58.1511185Z s3-bucket: gha-artifacts 2024-08-20T23:43:58.1511529Z env: 2024-08-20T23:43:58.1511815Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:58.1512279Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:58.1513032Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:58.1513688Z ##[endgroup] 2024-08-20T23:43:58.1540595Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:43:58.1541270Z # Remove any previous test jsons if they exist 2024-08-20T23:43:58.1541806Z rm -f test-jsons-*.zip 2024-08-20T23:43:58.1542348Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:43:58.1551625Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:58.1552139Z env: 2024-08-20T23:43:58.1552436Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:58.1552900Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:58.1553660Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:58.1554676Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773 2024-08-20T23:43:58.1555429Z ##[endgroup] 2024-08-20T23:43:58.1914765Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:43:58.1944581Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:43:58.1945465Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:43:58.1951551Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:43:58.1957211Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:43:58.2420563Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:43:58.2421234Z # Remove any previous test reports if they exist 2024-08-20T23:43:58.2421775Z rm -f test-reports-*.zip 2024-08-20T23:43:58.2422380Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:43:58.2431566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:58.2432078Z env: 2024-08-20T23:43:58.2432365Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:58.2432835Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:58.2433596Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:58.2434627Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773 2024-08-20T23:43:58.2435380Z ##[endgroup] 2024-08-20T23:43:58.2695315Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-08-20T23:43:58.2696231Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 86%) 2024-08-20T23:43:58.2697241Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 72%) 2024-08-20T23:43:58.2698157Z adding: test/test-reports/training_huggingface.csv (deflated 74%) 2024-08-20T23:43:58.2704807Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-08-20T23:43:58.2705801Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 74%) 2024-08-20T23:43:58.2738415Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:43:58.2739070Z # Remove any previous usage logs if they exist 2024-08-20T23:43:58.2739578Z rm -f logs-*.zip 2024-08-20T23:43:58.2740469Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:43:58.2741291Z # so check to see if the file exists first 2024-08-20T23:43:58.2741858Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:43:58.2742403Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:43:58.2742904Z fi 2024-08-20T23:43:58.2743276Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:43:58.2743868Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:43:58.2744366Z fi 2024-08-20T23:43:58.2752996Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:58.2753505Z env: 2024-08-20T23:43:58.2753794Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:58.2754253Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:58.2755010Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:58.2756033Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773 2024-08-20T23:43:58.2756790Z ##[endgroup] 2024-08-20T23:43:58.2849956Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:43:58.2937311Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:43:58.2938511Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:43:58.2939072Z rm -f debug-*.zip 2024-08-20T23:43:58.2939461Z if [ -d 'test/debug' ]; then 2024-08-20T23:43:58.2939966Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:43:58.2940447Z fi 2024-08-20T23:43:58.2948532Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:43:58.2949044Z env: 2024-08-20T23:43:58.2949322Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:58.2949787Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:58.2950539Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:58.2951549Z FILE_SUFFIX: test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773 2024-08-20T23:43:58.2952289Z ##[endgroup] 2024-08-20T23:43:58.3051475Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:58.3051943Z with: 2024-08-20T23:43:58.3052229Z s3-bucket: gha-artifacts 2024-08-20T23:43:58.3052678Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:58.3053164Z retention-days: 14 2024-08-20T23:43:58.3053509Z if-no-files-found: warn 2024-08-20T23:43:58.3053881Z path: test-jsons-*.zip 2024-08-20T23:43:58.3054239Z name: artifact 2024-08-20T23:43:58.3054553Z region: us-east-1 2024-08-20T23:43:58.3054858Z env: 2024-08-20T23:43:58.3055141Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:58.3055612Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:58.3056367Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:58.3057016Z ##[endgroup] 2024-08-20T23:43:58.6643580Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:43:58.6644198Z With the provided path, there will be 1 file uploaded 2024-08-20T23:43:58.6644831Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:58.7889243Z Starting upload of test-jsons-test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773.zip 2024-08-20T23:43:58.8786028Z Finished upload of test-jsons-test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773.zip 2024-08-20T23:43:58.9059932Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:58.9060384Z with: 2024-08-20T23:43:58.9060680Z s3-bucket: gha-artifacts 2024-08-20T23:43:58.9061123Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:58.9061634Z retention-days: 14 2024-08-20T23:43:58.9062000Z if-no-files-found: error 2024-08-20T23:43:58.9062381Z path: test-reports-*.zip 2024-08-20T23:43:58.9062744Z name: artifact 2024-08-20T23:43:58.9063050Z region: us-east-1 2024-08-20T23:43:58.9063560Z env: 2024-08-20T23:43:58.9063849Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:58.9064322Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:58.9065076Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:58.9065752Z ##[endgroup] 2024-08-20T23:43:59.2639895Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:43:59.2641504Z With the provided path, there will be 1 file uploaded 2024-08-20T23:43:59.2642410Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:59.2696516Z Starting upload of test-reports-test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773.zip 2024-08-20T23:43:59.3625760Z Finished upload of test-reports-test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773.zip 2024-08-20T23:43:59.3918804Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:59.3919255Z with: 2024-08-20T23:43:59.3919563Z s3-bucket: gha-artifacts 2024-08-20T23:43:59.3920007Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:59.3920486Z retention-days: 14 2024-08-20T23:43:59.3920828Z if-no-files-found: ignore 2024-08-20T23:43:59.3921493Z path: logs-*.zip 2024-08-20T23:43:59.3921842Z name: artifact 2024-08-20T23:43:59.3922168Z region: us-east-1 2024-08-20T23:43:59.3922474Z env: 2024-08-20T23:43:59.3922756Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:59.3923220Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:59.3923974Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:59.3924636Z ##[endgroup] 2024-08-20T23:43:59.7456895Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:43:59.7457527Z With the provided path, there will be 1 file uploaded 2024-08-20T23:43:59.7458156Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:59.7514750Z Starting upload of logs-test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773.zip 2024-08-20T23:43:59.8462059Z Finished upload of logs-test-dynamic_inductor_huggingface-1-1-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528773.zip 2024-08-20T23:43:59.8736704Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:43:59.8737158Z with: 2024-08-20T23:43:59.8737455Z s3-bucket: gha-artifacts 2024-08-20T23:43:59.8737904Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:43:59.8738390Z retention-days: 14 2024-08-20T23:43:59.8738735Z if-no-files-found: ignore 2024-08-20T23:43:59.8739118Z path: debug-*.zip 2024-08-20T23:43:59.8739441Z name: artifact 2024-08-20T23:43:59.8739748Z region: us-east-1 2024-08-20T23:43:59.8740061Z env: 2024-08-20T23:43:59.8740347Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:43:59.8740815Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:43:59.8741560Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:43:59.8742232Z ##[endgroup] 2024-08-20T23:44:00.2214906Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:44:00.2486464Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:44:00.2486984Z # shellcheck disable=SC2156 2024-08-20T23:44:00.2487943Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:44:00.2498145Z shell: /usr/bin/bash -e {0} 2024-08-20T23:44:00.2498512Z env: 2024-08-20T23:44:00.2498807Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:44:00.2499280Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:44:00.2500043Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:44:00.2501019Z ##[endgroup] 2024-08-20T23:44:00.5121462Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:44:00.5122072Z with: 2024-08-20T23:44:00.5122337Z env: 2024-08-20T23:44:00.5122632Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:44:00.5123094Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:44:00.5123852Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:44:00.5124517Z ##[endgroup] 2024-08-20T23:44:00.5144124Z ##[group]Run set -eou pipefail 2024-08-20T23:44:00.5144540Z set -eou pipefail 2024-08-20T23:44:00.5144897Z  2024-08-20T23:44:00.5145661Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:44:00.5146307Z for _ in $(seq 1440); do 2024-08-20T23:44:00.5146791Z  # Break if no ssh session exists anymore 2024-08-20T23:44:00.5147299Z  if [ "$(who)" = "" ]; then 2024-08-20T23:44:00.5147711Z  break 2024-08-20T23:44:00.5148056Z  fi 2024-08-20T23:44:00.5148361Z  echo "." 2024-08-20T23:44:00.5148695Z  sleep 5 2024-08-20T23:44:00.5149021Z done 2024-08-20T23:44:00.5157673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:44:00.5158173Z env: 2024-08-20T23:44:00.5158454Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:44:00.5158917Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:44:00.5159876Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:44:00.5160526Z ##[endgroup] 2024-08-20T23:44:00.5189505Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:44:00.5316291Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:44:00.5317063Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:44:00.5317670Z # shellcheck disable=SC2046 2024-08-20T23:44:00.5318149Z docker stop $(docker ps -q) || true 2024-08-20T23:44:00.5318633Z # Prune all of the docker images 2024-08-20T23:44:00.5319096Z docker system prune -af 2024-08-20T23:44:00.5327685Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:44:00.5328194Z env: 2024-08-20T23:44:00.5328479Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:44:00.5328947Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:44:00.5329706Z DOCKER_CONTAINER_ID: b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:44:00.5330360Z ##[endgroup] 2024-08-20T23:44:01.0997065Z b7a452073000 2024-08-20T23:44:01.8445955Z Deleted Containers: 2024-08-20T23:44:01.8446472Z b7a452073000071b4bfca03115e5a7c0cbe6e31bc242bfddbd4e2ba01dce6450 2024-08-20T23:44:01.8446919Z 2024-08-20T23:44:12.8073752Z Deleted Images: 2024-08-20T23:44:12.8075807Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:44:12.8078775Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:44:12.8080680Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:44:12.8081980Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:44:12.8083054Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:44:12.8083895Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:44:12.8084714Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:44:12.8085577Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:44:12.8086491Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:44:12.8087720Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:44:12.8088565Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:44:12.8089438Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:44:12.8090322Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:44:12.8091151Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:44:12.8092199Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:44:12.8093252Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:44:12.8094089Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:44:12.8094932Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:44:12.8095886Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:44:12.8096725Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:44:12.8097570Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:44:12.8098404Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:44:12.8099468Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:44:12.8100606Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:44:12.8101569Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:44:12.8102399Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:44:12.8103227Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:44:12.8104078Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:44:12.8104960Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:44:12.8105806Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:44:12.8106649Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:44:12.8107482Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:44:12.8108336Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:44:12.8109171Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:44:12.8110054Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:44:12.8110884Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:44:12.8111729Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:44:12.8112569Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:44:12.8113405Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:44:12.8114240Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:44:12.8115077Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:44:12.8115930Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:44:12.8116777Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:44:12.8117636Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:44:12.8118471Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:44:12.8119293Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:44:12.8120343Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:44:12.8121481Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:44:12.8122335Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:44:12.8123158Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:44:12.8123998Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:44:12.8124854Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:44:12.8125690Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:44:12.8126513Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:44:12.8127348Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:44:12.8128209Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:44:12.8129059Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:44:12.8129910Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:44:12.8130745Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:44:12.8131719Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:44:12.8132547Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:44:12.8133375Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:44:12.8134233Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:44:12.8135099Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:44:12.8135956Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:44:12.8136788Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:44:12.8137619Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:44:12.8138456Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:44:12.8139293Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:44:12.8140134Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:44:12.8140988Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:44:12.8141829Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:44:12.8142662Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:44:12.8143501Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:44:12.8144371Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:44:12.8145215Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:44:12.8146074Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:44:12.8146937Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:44:12.8147812Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:44:12.8148686Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:44:12.8149541Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:44:12.8150047Z 2024-08-20T23:44:12.8150204Z Total reclaimed space: 39.92GB 2024-08-20T23:44:12.8228917Z Post job cleanup. 2024-08-20T23:44:12.8280520Z Post job cleanup. 2024-08-20T23:44:12.9157491Z [command]/usr/bin/git version 2024-08-20T23:44:12.9210113Z git version 2.40.1 2024-08-20T23:44:12.9247586Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c90b86cf-5f09-43a5-b7ae-d4d25e8f0001' before making global git config changes 2024-08-20T23:44:12.9248864Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:44:12.9252636Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:44:12.9300574Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:44:12.9344711Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:44:12.9715148Z Entering 'android/libs/fbjni' 2024-08-20T23:44:12.9785679Z Entering 'third_party/FP16' 2024-08-20T23:44:12.9854129Z Entering 'third_party/FXdiv' 2024-08-20T23:44:12.9921070Z Entering 'third_party/NNPACK' 2024-08-20T23:44:12.9992418Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:44:13.0059450Z Entering 'third_party/XNNPACK' 2024-08-20T23:44:13.0143654Z Entering 'third_party/benchmark' 2024-08-20T23:44:13.0212398Z Entering 'third_party/cpp-httplib' 2024-08-20T23:44:13.0279888Z Entering 'third_party/cpuinfo' 2024-08-20T23:44:13.0353420Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:44:13.0421536Z Entering 'third_party/cutlass' 2024-08-20T23:44:13.0496178Z Entering 'third_party/eigen' 2024-08-20T23:44:13.0565952Z Entering 'third_party/fbgemm' 2024-08-20T23:44:13.0635367Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:44:13.0699495Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:44:13.0767138Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:44:13.0842442Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:44:13.0915636Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:44:13.0986624Z Entering 'third_party/flatbuffers' 2024-08-20T23:44:13.1058356Z Entering 'third_party/fmt' 2024-08-20T23:44:13.1126465Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:44:13.1194832Z Entering 'third_party/gloo' 2024-08-20T23:44:13.1264155Z Entering 'third_party/googletest' 2024-08-20T23:44:13.1332566Z Entering 'third_party/ideep' 2024-08-20T23:44:13.1397558Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:44:13.1476221Z Entering 'third_party/ittapi' 2024-08-20T23:44:13.1548086Z Entering 'third_party/kineto' 2024-08-20T23:44:13.1614694Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:44:13.1681543Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:44:13.1750365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:44:13.1823623Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:44:13.1890968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:44:13.1961136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:44:13.2040708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:44:13.2109692Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:44:13.2176172Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:44:13.2245114Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:44:13.2315514Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:44:13.2382559Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:44:13.2453048Z Entering 'third_party/mimalloc' 2024-08-20T23:44:13.2522129Z Entering 'third_party/nccl/nccl' 2024-08-20T23:44:13.2589519Z Entering 'third_party/nlohmann' 2024-08-20T23:44:13.2659528Z Entering 'third_party/onnx' 2024-08-20T23:44:13.2745297Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:44:13.2812529Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:44:13.2885058Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:44:13.2953600Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:44:13.3023115Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:44:13.3090775Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:44:13.3159853Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:44:13.3232619Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:44:13.3303372Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:44:13.3369318Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:44:13.3435117Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:44:13.3505452Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:44:13.3579017Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:44:13.3673798Z Entering 'third_party/pocketfft' 2024-08-20T23:44:13.3745148Z Entering 'third_party/protobuf' 2024-08-20T23:44:13.3813175Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:44:13.3880025Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:44:13.3950205Z Entering 'third_party/psimd' 2024-08-20T23:44:13.4018183Z Entering 'third_party/pthreadpool' 2024-08-20T23:44:13.4083703Z Entering 'third_party/pybind11' 2024-08-20T23:44:13.4151406Z Entering 'third_party/python-peachpy' 2024-08-20T23:44:13.4219295Z Entering 'third_party/sleef' 2024-08-20T23:44:13.4285596Z Entering 'third_party/tensorpipe' 2024-08-20T23:44:13.4352499Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:44:13.4419886Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:44:13.4486718Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:44:13.4553979Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:44:13.4617136Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:44:13.4713686Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:44:13.4750201Z http.https://github.com/.extraheader 2024-08-20T23:44:13.4759849Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:44:13.4806594Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:44:13.5171967Z Entering 'android/libs/fbjni' 2024-08-20T23:44:13.5215503Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5259670Z Entering 'third_party/FP16' 2024-08-20T23:44:13.5303920Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5347439Z Entering 'third_party/FXdiv' 2024-08-20T23:44:13.5390930Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5434854Z Entering 'third_party/NNPACK' 2024-08-20T23:44:13.5478202Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5522348Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:44:13.5565811Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5610694Z Entering 'third_party/XNNPACK' 2024-08-20T23:44:13.5654047Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5714255Z Entering 'third_party/benchmark' 2024-08-20T23:44:13.5758128Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5803103Z Entering 'third_party/cpp-httplib' 2024-08-20T23:44:13.5846554Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5889866Z Entering 'third_party/cpuinfo' 2024-08-20T23:44:13.5934916Z http.https://github.com/.extraheader 2024-08-20T23:44:13.5979534Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:44:13.6023203Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6066855Z Entering 'third_party/cutlass' 2024-08-20T23:44:13.6110977Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6162997Z Entering 'third_party/eigen' 2024-08-20T23:44:13.6206614Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6252259Z Entering 'third_party/fbgemm' 2024-08-20T23:44:13.6295986Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6341242Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:44:13.6388324Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6432332Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:44:13.6475252Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6521428Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:44:13.6568524Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6619043Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:44:13.6662425Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6708599Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:44:13.6752448Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6798611Z Entering 'third_party/flatbuffers' 2024-08-20T23:44:13.6847563Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6893621Z Entering 'third_party/fmt' 2024-08-20T23:44:13.6938863Z http.https://github.com/.extraheader 2024-08-20T23:44:13.6982580Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:44:13.7027379Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7071582Z Entering 'third_party/gloo' 2024-08-20T23:44:13.7116069Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7159583Z Entering 'third_party/googletest' 2024-08-20T23:44:13.7204378Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7247928Z Entering 'third_party/ideep' 2024-08-20T23:44:13.7291248Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7332219Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:44:13.7374679Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7426708Z Entering 'third_party/ittapi' 2024-08-20T23:44:13.7469793Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7513188Z Entering 'third_party/kineto' 2024-08-20T23:44:13.7556027Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7598196Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:44:13.7641885Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7683859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:44:13.7727957Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7773493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:44:13.7816889Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7863803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:44:13.7906163Z http.https://github.com/.extraheader 2024-08-20T23:44:13.7951264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:44:13.7993938Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8037162Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:44:13.8080471Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8129618Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:44:13.8172114Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8217065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:44:13.8260139Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8308380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:44:13.8351660Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8426571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:44:13.8445088Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8492226Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:44:13.8535609Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8579040Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:44:13.8622754Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8668714Z Entering 'third_party/mimalloc' 2024-08-20T23:44:13.8712921Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8756594Z Entering 'third_party/nccl/nccl' 2024-08-20T23:44:13.8801933Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8845772Z Entering 'third_party/nlohmann' 2024-08-20T23:44:13.8889466Z http.https://github.com/.extraheader 2024-08-20T23:44:13.8934532Z Entering 'third_party/onnx' 2024-08-20T23:44:13.8978487Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9034671Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:44:13.9077683Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9120550Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:44:13.9168161Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9216622Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:44:13.9259570Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9308941Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:44:13.9350908Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9393179Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:44:13.9441008Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9485255Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:44:13.9527859Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9569861Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:44:13.9612738Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9657177Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:44:13.9699591Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9741976Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:44:13.9784227Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9826592Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:44:13.9869100Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9909979Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:44:13.9952190Z http.https://github.com/.extraheader 2024-08-20T23:44:13.9997022Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:44:14.0041618Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0088115Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:44:14.0131454Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0197440Z Entering 'third_party/pocketfft' 2024-08-20T23:44:14.0242109Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0286037Z Entering 'third_party/protobuf' 2024-08-20T23:44:14.0330341Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0376099Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:44:14.0419636Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0466217Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:44:14.0509503Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0557371Z Entering 'third_party/psimd' 2024-08-20T23:44:14.0601502Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0646022Z Entering 'third_party/pthreadpool' 2024-08-20T23:44:14.0689672Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0733445Z Entering 'third_party/pybind11' 2024-08-20T23:44:14.0776582Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0819843Z Entering 'third_party/python-peachpy' 2024-08-20T23:44:14.0862440Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0905703Z Entering 'third_party/sleef' 2024-08-20T23:44:14.0948583Z http.https://github.com/.extraheader 2024-08-20T23:44:14.0992977Z Entering 'third_party/tensorpipe' 2024-08-20T23:44:14.1036435Z http.https://github.com/.extraheader 2024-08-20T23:44:14.1078623Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:44:14.1120992Z http.https://github.com/.extraheader 2024-08-20T23:44:14.1163798Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:44:14.1206124Z http.https://github.com/.extraheader 2024-08-20T23:44:14.1248629Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:44:14.1291188Z http.https://github.com/.extraheader 2024-08-20T23:44:14.1334707Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:44:14.1376991Z http.https://github.com/.extraheader 2024-08-20T23:44:14.1421966Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:44:14.1464864Z http.https://github.com/.extraheader 2024-08-20T23:44:14.1625231Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:44:14.1646956Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:44:14.1655014Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:44:14.1655550Z ##[endgroup] 2024-08-20T23:44:20.9859673Z Cleaning up orphan processes